Linux
Ricerca & Sviluppo
slogo2.gif (5059 byte)

pagina
principale

Linux aiuta a fare rivivere il Titanic

Pubblicato per la prima volta sul Linux Journal edizione #46


Immagine generata da computer del Titanic

Digital Domain utilizza Linux per creare effetti visuali ad alta tecnologia.

Articolo di Daryll Strauss tradotto in italiano da Franco Tronnolone, Nuvoli Massimo.

Digital Domain è uno studio di produzione avanzato che si trova in California a Venice. In questo studio vengono creati effetti visivi da inserire in film, in prodotti commerciali e in applicazioni multimediali. Tra le produzioni di film che può vantare lo studio possono essere citati Intervista con il Vampiro, True Lies, Apollo 13, Dante's Peak e Il Quinto Elemento. Per ciò che concerne invece i prodotti commerciali l'elenco sarebbe troppo lungo. (è disponibile presso il sito web http://www.d2.com). Questo studio è sicuramente conosciuto non solo per la eccellente qualità del suo lavoro, ma anche per il contributo creativo apportato.

Il film Titanic (scritto e diretto da James Cameron) è stato presentato nei cinema il 19 dicembre 1997. Girato sul Titanic durante la sua prima ed ultima traversata dell'oceano Atlantico, questa storia doveva essere ricreata sullo schermo in tutto lo splendore della nave e il dramma della tragedia. Digital Domain è stato selezionato per produrre un gran numero di effetti video difficili per l'esigenza di questo film.

L'Operazione

Gli effetti digitali video sono una grande parte del nostro lavoro. Per molte sequenze di effetti digitali, le immagini fotografiche originali sono per prima riprese sulla pellicola (usando i metodi cinematografici convenzionali) e quindi sono esportate nel calcolatore. Ciascun "fotogramma" o "scena" è installata come una serie di directory con una directory "elemento" per tutti i passaggi fotografici che contribuiscono alla scena finale. Ogni struttura della pellicola è memorizzata come un file separato su un server di file centrale. Un artista digitale allora comincia a lavorare sulla sequenza. Il lavoro può svilupparsi creando nuovi elementi interi come l'animazione e la rappresentazione di modelli 3D o modificando gli elementi attuali come la verniciatura fuori una linea o isolando le zone di interesse nella pellicola originale. Questo lavoro è fatto sul desktop dell artista (spesso su NT workstation o SGI). Una volta che la messa a punto di questo lavoro è fatta, il processo è ripetuto per ogni struttura della sequenza. Questa elaborazione sequenziale è fatta su tutte le PCUs disponibili in modo facile, spesso parallelamente e richiede un file system distribuito e una uniforme veduta dei dati. Un obiettivo di questo processo è di rimanere indipendente dalla piattaforma per quanto possibile. Per concludere, una volta che tutti gli elementi sono creati, l'immagine finale è "composta". Durante questo punto i diversi elementi sono corretti da colore per abbinare la fotografia originale, coordinato e stratificato nello spazio per creare l'immagine finale. Di nuovo, la messa a punto del lavoro di composizione è fatta solitamente sul desktop SGI e L'elaborazione sequenziale è fatta in tutta facilità.

Poichè sviluppare un modello in grande scala del Titanic sarebbe stato proibitivamente costoso, solo una parte della nave fu costruita nella misura grande (dal personale di produzione) e le miniature sono state usate per il resto delle scene. A questo modello abbiamo aggiunto altri elementi della scena quali L'oceano, la gente, gli uccelli, il fumo ed altri particolari che fanno sembrare essere attraccata, in navigazione o affondare nell'oceano. Inoltre, abbiamo sviluppato un modello 3D ed abbiamo fotografato elementi 2D per simulare la fotografia subacquea, aereo-trasportata ed a terra.

Durante il lavoro sul Titanic la facilità è stata raggiunta approssimativamente da 350 SGI CPUs, 200 DEC Alpha CPUs e 5 terabyte di disco tutti connessi tramite una faster network da 100MBps.

L'Analisi

Il nostro obiettivo è sempre creare immagini di più alta qualità all'interno dei vincoli di programma e finanziari. La creazione di immagini è compiuta in due fasi. Nella prima fase, l'artista digitale lavora ad una workstation interattiva che utilizza i pacchetti di programmi specifici e specializzati e l'hardware ad alto rendimento specifico. Durate la seconda fase il lavoro è elaborato in sequenze su altrettanto PCUs come possibile, senza rigurado al periodo d'installazione, alla posizione o alle caratteristiche per aumentare le prestazioni interattive. E' difficile da migliorare su quella prima fase interattiva. Gli artisti digitali richiedono determinati pacchetti che non sono sempre disponibili su altre piattaforme. Anche se simili pacchetti sono disponibili, c'è un costo significativo connesso all'interoperatività fra loro. Un altro problema è che alcuni dei pacchetti richiedono spesso una determinazione high-end dell'hardware (spesso 3D). Le stesse qualità e prestazioni di accelirazioni 3D non possono essere disponibili su altre piattaforme. Nella fase elaborazione-sequenza-sequenza, si sono trovati più facilmente dei miglioramenti, poichè i requisiti di base sono il calcolo di alta-larghezza di banda, l'accesso a grande memoria una rete veloce. Se sono disponibili applicazioni adatte, possiamo migliorare quella parte del processo. Anche nei casi in cui è disponibile soltanto un sottinsieme delle applicazioni per una particolare piattaforma, usando quella piattaforma ci dà la capacità di dividere il flusso del lavoro per migliorare l'accesso alle risorse in generale. Abbiamo concluso velocemente che i sistemi DEC Alpha- based hanno risposto molto bene alle nostre esigenze di elaborazione-sequenza-sequenza. Forniscono prestazioni di virgola mobile estremamente alte nella confezione del prodotto. Potevamo identificare determinate applicazioni floating-punto-intense come meta dell obiettivo.


La traduzione di questo testo è in corso: questo è il LIMITE DI TRADUZIONE, Il testo al di sotto di questa separazione è ancora quello originale in inglese.
Se sei interessato a questo articolo tradotto puoi lasciarmi un messaggio sarò lieto di avvisarti di ogni aggiornamento.


Digital UNIX performed very well and integrated nicely into our environment. The biggest limitations of Digital UNIX were cost and lack of flexibility. We would be purchasing and reconfiguring a large number of systems. Separately purchasing Digital UNIX for each system would have been time consuming and expensive. Digital UNIX also didn't have certain extensions we required and could not provide them in an acceptable time frame. For example, we needed to communicate with our NT-based file servers, connect two unusual varieties of tape drives and allow large numbers of users on a single system; none are supported by Digital UNIX. Linux fulfilled the task very well. It handled every job we threw at it. During our testing phase, we used its ability to emulate Digital UNIX applications to benchmark standard applications and show that its performance would meet our needs. The flexibility of the existing devices and available source code gave Linux a definitive advantage. The downside of Linux was the engineering effort required to support it. We knew that we would need to dedicate one engineer to support these systems during their set up. Fortunately, we had engineers with significant previous experience with Linux on Intel systems (the author and other members of the system-administration staff) and enough Unix-system experience to make any required modifications. We carefully tested a variety of hardware to make sure all were completely compatible with Linux.

Numerology and Fate

The Linux distribution used was Red Hat 4.1. At that time Red Hat was shipping Linux 2.0.18, which didn't support the PC164 mainboard, so the first thing we had to do was upgrade the kernel. During our testing we tracked down a number of problems with devices and kept up with both the 2.0 and 2.1 series of kernels. We ended up sticking with 2.1.42 with a few patches. We also decided on the NCR 810 SCSI card with the BSD-based driver and the SMC 100MB Ethernet card with the de4x5 driver. It turned out to be a very stable configuration, but there was one serious floating-point problem that caused our water-rendering software to die with an unexpected floating-point exception. This turned out to be a tricky problem to fix and didn't make it into the kernel sources until 2.0.31-pre5 and 2.1.43. The Alpha kernel contains code to catch floating-point exceptions and to handle them according to the IEEE standard. That code failed to handle one of the floating-point instructions that could generate an exception. As a result, when that case occurred, the application would exit with a floating-point exception. Once fixed, our applications ran quite smoothly on the Alpha systems.

The Implementation

At this point the decision was made to purchase 160 433MHz DEC Alpha systems from Carrera Computers of Newport Beach, California. Of those 160 machines, 105 of the machines are running Linux, the other 55 are running NT. The machines are connected with 100Mbps Ethernet to each other and to the rest of our facility.

The staff at Carrera was extraordinarily helpful and provided inestimable support for our project. This support began at the factory, with follow-up support through delivery, support and repair. We created a master disk, which we provided to Carrera, along with a single initialization script that would configure the generic master disk to one of the 160 unique personalities by setting up parameters such as the system name and IP address. Carrera built, configured and burned-in the machine, then logged in as a special user causing the setup script to execute. When the script completed, the machine automatically shut down. This process made configuring the machines easy for both Carrera and us. When the hosts arrived, we just plugged them in and flipped the switch, and they came up on the network. All 160 machines are housed in a small room at Digital Domain in ten 19 inch racks. They are all connected to a central screen, keyboard and mouse via a switching system to allow an operator to sit in the middle of the room and work on the console of any machine in the room. Figure 1. Digital Domain Computer Room The room was assembled in a time period of two weeks including the installation of the electrical, computing and networking. The time spent creating the initialization script was extremely well spent as it allowed the machines to be dropped in place with relatively little trouble. At that point we began running the Titanic work through the "Render Ranch" of Alphas. The first part of this work partition was to simulate and render the water elements. We knew that the water elements were computationally very expensive, so this process was one of the major reasons for purchasing the Alphas. These jobs computed for approximately 45 minutes and then generated several hundred megabytes of image data to be stored on central storage servers. Intermediate data was stored on the local SCSI disk of the Alpha. The floating-point power of the DEC Alpha made jobs run about 3.5 times faster than on our old SGI systems. As the water rendering completed, the task load then switched to compositing. These jobs were more I/O bound, because they had to read elements from disks on servers spread around the facility and combine them into frames to be stored centrally. Even so, we still saw improvements of a factor of two for these tasks. We were extremely pleased with the results. Between the beginning of June and the end of August, the Alpha Linux systems processed over three hundred thousand frames. The systems were up and running 24 hours a day, seven days a week. There were no extended downtimes, and many of the machines were up for more than a month at a time.

The Problems

We addressed a number of different problems using a variety of techniques. Some of the problems were Alpha specific, and some were issues for the Linux community at large. Hopefully, these issues will help others in the same position and provide feedback for the Linux community. Hardware compatibility, particularly with Alpha Linux, is still a problem. Carrera was very cooperative about sending us multiple card varieties, so that we could do extensive testing. The range of choices was large enough that we were able to find a combination that worked. We had to pay careful attention to which products we were using, as the particular chip revision made a difference in one case. The floating-point problem (discussed above) was the toughest problem we had to address. We didn't expect to find this kind of problem when we started the project. This was a long-standing bug that had never been tracked down--we attribute this fact to the relatively small Alpha Linux community. Linux software for Alpha seems to be less tested than the equivalent software for the Intel processors--again, a function of the user-base size. It was exacerbated by the fact that Alpha Linux uses glibc instead of libc5, which introduced problems in our code and, we suspect, in other packages. We had a number of small configuration issues with respect to the size of our facility. Most of these were just parameter changes in the kernel, but they took some effort to track down. For example, we had to increase the number of simultaneously mounted file systems (64 was not sufficient). Also, NFS directory reads were expected to fit within one page (4K on Intel, 8K on Alpha); we had to double this number to support the average number of frames stored in a single directory. Boot management under Linux Alpha was more difficult than we would have liked. We felt the documentation needed improvements to make it more useful. Boot management required extensive knowledge of ARC, MILO and Linux to make it work. ARC requires entering a reasonably large amount of data to get MILO to boot. MILO worked well and provided a good set of options, but we never managed to get soft reboots to operate correctly. We've been working with the engineers at DEC to improve some of these issues. The weakest link in the current Linux kernel appeared to be the NFS implementation, resulting in most of our system crashes. We generally had a large number of file systems mounted simultaneously, and those file systems were often under heavy load. When central servers died or had problems, the Linux systems didn't recover. The common symptoms of these problems were stale NFS handles and kernel hangs. When all the servers were running, the Linux boxes worked correctly. Overall, the NFS implementation worked, but it should be more robust.

Conclusion

The Linux systems worked incredibly well for our problems. The cost benefit was overwhelmingly positive even including the engineering resources we devoted to the problems. The Alpha Linux turned out to be slightly more difficult than first expected, but the state of Alpha Linux is improving very rapidly and should be substantially better now. Digital Domain will continue to improve and expand the tools we have available on these systems. We are engendering the development of more commercial and in-house applications available on Linux. We are requesting that vendors port their applications and libraries. At this time, the Linux systems are only used for batch processing, but we expect our compositing software to be used interactively by our digital artists. This software does not require dedicated acceleration hardware, and the speed provided by the Alpha processor is a great benefit to productivity. Feature film and television visual effects development has provided a high-performance, cost-sensitive, proving ground for Linux. We believe that the general purpose nature of the platform coupled with commodity pricing gives it wide application in areas outside our industry. The low entry cost, versatility and interoperability of Linux is sufficiently attractive to warrant more extensive investigation, experimentation and deployment. We are currently at the forefront of that development within our industry and hope to be joined shortly by our peers.

Why Risk Linux? A Production Perspective 

by Wook 

Currently, Digital Domain's core business is as a premier provider of visual effects creativity and services to the feature film and commercial production industries. As such, we often take a conservative approach to changes in infrastructure and methodologies in order to meet aggressive delivery schedules and the most demanding standards of product quality. 

During the course of work on several recent feature film productions, we encountered situations where our installed base of equipment was not adequate to meet changing production schedules and dynamic visual effects requirements (in terms of increasing magnitude of effort and complexity). We needed to meet these challenges head on without impacting the existing pipeline and without creating new methodologies or systems which would require re-engineering or re-training. Linux Alpha helped us overcome these challenges both cost effectively and quickly (a rare combination). 

Selecting Linux as part of the production pipeline for the film Titanic required several goals to be met. If we had not met these requirements, it is unlikely we would have been able to deliver sufficient computing resources in a timely fashion to the production. We needed interoperability and, to a certain degree, compatibility with our SGI/Irix-based systems. Interoperability and compatibility with Linux had been demonstrated during a previous effort (Dante's Peak). We ported critical infrastructure elements (to support distributed processing) to the Linux environment in days, not weeks, using existing staff. The developers of these tools were able to rapidly deploy to the Linux environment, demonstrating that we could leverage that environment in short order. We needed performance, as the schedule for the production, as well as the magnitude of the work implied a 100% or more increase in studio processing capacity. As we had shown that Alpha Linux provided a factor of three to four over our SGI systems (see main article), it was possible to deliver that increased level of performance while physically constrained (air, power and floor space) within our current facility. 

As to cost effectiveness, we would have needed more than twice as many Intel machines as Alphas to meet our performance goals. SGI was a valid contender, but could not compete on a price per CPU basis. We also needed a viable structure for delivery, installation and support. Carrera Computers had proven their ability to supply and support us in a timely and cost-effective manner prior to this order, and that company continued to provide an extraordinary level of service throughout the Titanic project. 

All things considered, this risk paid off in substantial dividends of project quality and time. Because the urgency of the situation demanded that we think "outside the box", we were able to deliver a superior solution in a framework that was entirely compatible with our normal operating models and that gave a productivity increase equal to double that of our previous infrastructure. The satisfaction in this success actually made up for the stress incurred in risking one's job and career.

Daryll Strauss is a software engineer at Digital Domain. He has been hacking on Unix systems of one variety or another for the last 15 years, but he gets the most enjoyment out of computer graphics. He has spent the last five years working in the film industry doing visual effects for film. He can be reached at daryll@d2.com.

Wook has been a software engineer for over 20 years, having discovered computers and became a complete geek at the age of 14. He has worked for many companies over those years, finally coming to rest at Digital Domain, where he was considered unfit for the task of software engineering and has been relegated to the position of Director of (Digital) Engineering.


Accessi a questa pagina (con interni ) ultima modifica 18-05-1998 19:31