hledat:

Architecture of CineGrid Node in the Czech Republic

CESNET technical report 24/2008
PDF format

Michal Krsek

Received 31. 12. 2008

Abstract

This technical report describes Czech node of CineGrid node built by CESNET. We have three different displaying units as well as three different sending platforms provided by CESNET and its partners. We have built CineGrid Node as distributed platform, devices are interconnected by high speed photonic network Prague MediaNet. Czech research and academic industry has platform for cooperation with Czech and global filmmaking industry.

Keywords: digital cinematography, CineGrid, network media

1  Architecture of CineGrid node within the Czech Republic

CineGrid is a non-profit international membership organization administratively based in California. CineGrid has been established to promote research, development and deployment of new distributed applications of ultra-high performance digital media (sound and picture) over advanced networks, using Grid computing technologies for networked collaboration.

CESNET is network member of CineGrid, participating actively in CineGrid working groups and working on CineGrid demonstrations. CESNET has two areas of focus in CineGrid – collaboration in rich media high-quality environment and cooperation with film industry that has great tradition in the Czech Republic.

To strengthen its position in CineGrid community and create better conditions for local creative community, CESNET has decided to build distributed CineGrid node – a set of devices capable of working with high-quality media interconnected by high speed photonic network called Prague MediaNet.

2  Prague MediaNet

Prague MediaNet is transparent high-speed optical infrastructure with backbone links at 10 Gbps speed. Users can be connected by 1 Gbps or 10 Gbps connections. Before connection to Prague MediaNet potential user has to demonstrate its ability to work with high quality rich media (e.g. having devices able to work with HD audio and video) and comply CESNET's Access Policy.

Basic infrastructure of Prague MediaNet is network infrastructure that permanently interconnects CESNET premises at Zikova 4 with Barrandov campus. This network is connected to rest of CESNET2 network on end-to-end (either VLANs or dedicated lightpaths) or IP/IPv6 based connections. Prague MediaNet has connection to hybrid optical research networks through CESNET2 lightpaths to Chicago and Amsterdam or via GEANT2+ services for reach European Research Area.

Direct connection to Prague MediaNet infrastructure is possible directly in Barrandov campus, CESNET2 PoPs in Prague (Zikova 4 or Congress centre). Indirectly, users can use CESNET2 optical network, academic metropolitan networks or local networks at CESNET members.

Indirect permanent connection to Prague MediaNet has Laboratory of Advanced Network Technologies (Sitola), a joint effort of Faculty of Informatics and Institute of Computer Science of Masaryk University and CESNET, based in Brno.

3  Display and projector

3.1  SONY SRX

As one of the focus of CESNET activity is collaboration with film industry, we have access to screening room/small cinema at CINEPOST at Barrandov Studios campus. This room is equipped by SONY SRX 4K projector with native resolution 4096x2160 pixels (4K). This projector can be equipped by four DVI-I interface cards or four dual-link HD-SDI (SMPTE 294M/292M/296M/372M) cards. Supported frame rates are 24, 48 and 60 fps, so both cinema and TV content can be displayed. Projector is able to display other resolutions, including HD (1920x1080).

This projector is fixed in the room and used for film-postprocessing purposes, thus it is not possible to transport it for event.

3.2  AstroDesign DM-3400

For demonstrations and research & development outside the cinema we have 56" monitor Astro Systems DM-3400. This monitor has both DVI-D and dual link HD-SDI inputs. Display resolution is 3840x2160 pixels (SHD / Super HD), with upgrade of firmware scheduled to January 2009 to support 4K. Frame rates 24 and 60 fps are available. Monitor accepts HD-SDI embedded audio input in first HD-SDI interface.

Monitor itself needs to have synchronised all inputs. It is relatively easy to feed it via HD-SDI interfaces, because whole HD-SDI infrastructure is usually synchronized. There is a problem with getting synchronized signal using PC platform with DVI interfaces (solution is described bellow).

This monitor is moveable, but due of its weight 45 kg needs it takes two to four people to arrange it.

4  Processing units

4.1  NTT 4k JPEG 2000 codec

We have NTT JPEG 2000 codec for playing JPEG2000 streams from hard drives or from network. This device has two logical parts. Rack mounted 4U high PC and JPEG 2000 encoding/decoding boards with HD-SDI (SMTPE 372M) outputs. Codec has two 10/100/1000 base ports and control interfaces (keyboard, VGA, RS-232). Codec supports 4K, SHD and HD modes at frame rates 24, 30 and 60 fps, Codec is Linux based, drivers for JPEG 2000 boards are present in binary form only.

Codec has five modes of operation:

1. Real time transmission – codec receives HD-SDI signal thru JPEG 2000 cards, encodes it (on cards) and send encoded to the network. Usually used encoding profile has bit rate ranging from 250 to 700 Mb/s.
2. Live encoding – codec receives HD-SDI signal through JPEG 2000 cards, encodes it (using JPEG 2000 cards) and stores JPEG 2000 files on hard drives.
3. Batch file processing – codec reads files from hard drives, encodes them (using software only) and stores them encoded back to hard drives.
4. VoD (Video on Demand) – codec reads encoded files from drives and send the stream to the network to another, playing codec.
5. Playout – codec reads files from drives or data stream from network and decodes these data on cards and sends them to output device (connected to cards).

4.2  UltraGrid / iHDTV systems

For low latency transmission of uncompressed HD over network we use open source software UltraGrid[2] and iHDTV. UltraGrid software is Linux/MacOS X based while iHDTV is Windows based software. As UltraGrid is being co-developed by our colleagues at Sitola, there are works on implementation of 2K video and has compatibility mode for iHDTV [2], we focus more on use of UltraGrid.

We have several semi-dedicated PCs that are being used as UltraGrid / iHDTV hosts in case of need.

4.3  SAGE renderer

For display only we have machine equipped by SAGE (Scalable Adaptive Graphics Environment) a system that is primary focused to visualize scientific data on tiled displays. There is tiled display used at Sitola that is used mostly for visualization. As we want to support cinema industry and special display devices (as Sony SRX or Astro Systems DM-3400), we had to build special configuration of SAGE than fits to our needs.

We decided to build a system that is able to run in cinema like environment, with or without network and focused on play out 4K and SHD content on our screens. We had to design silent PC, with four synchronized DVI outputs, having 1GE and 10GE network interfaces and relatively large and fast disk subsystem. Such as system is not on the market – typical high computing power machines are a noisier than is acceptable. So at the beginning we decided to have water cooled system and we needed to choose all cards water cooling capable.

4.3.1  Hardware

As we needed lot of functionalities, we opted for a motherboard with at least four PCIe x8 slots (two graphics cards, 10 GE network card and RAID controller). At the time of building of this encoder there was no board supporting this request, so we have chosen SuperMicro H8DAE-2 motherboard. It has two PCIe x16, one PCIe x8 and one PCIe x4 ports. This board is equipped with two AMD Opteron 2220 - 2,8 GHz processors and eight 2GB DDR2 memory banks.

Disk subsystem has five 1TB disks (1 system, four data in RAID 0) connected to RAID controller 3WARE 9650SE (in PCIe x4 slot). We have also one 32GB SSD drive used for caching.

The machine is connected to the network via two 10/100/1000 network adapters integrated on-board or via Myrinet 10GE network card (in PCIe x8 slot) with XFP interface. All network interfaces have 9K frame size.

Graphics cards are most complicated part to find and integrate. We needed four DVI or HD-SDI outputs capable to display HD resolution. These four outputs must have frame synchronization. The only solution we have found was NVIDIA Quadro FX 4600/5600 family with NVIDIA Quadro G-Sync II synchronization card. As we do not need top level rendering speed, we have chosen two NVIDIA Quadro FX 4600 cards (in PCIe x16 slots) with one NVIDIA Quadro G-Sync II card.

The machine was to be transportable, thus we have decided to use midi tower case. Although we have used water cooling, we needed to create extra holes into the case to help the air flow. These holes have attached low speed fans that aren’t noisy.

4.3.2  Configuration

We installed the machine with the 64bit Linux distribution Ubuntu 8.04. There was nothing special in the settings during installation (just installed workstation with packages we’d like to have).

After installation of X Window system we had to download and install latest linux drivers from nVidia site. After that we had to set-up all screens in X-Window and synchronize frame lock. This can be done using GUI interface in program nvidia-settings. As the system doesn’t preserve configuration (even if you click “save" option), we recommend to set the configuration using nvidia-settings hand written configuration file or to use CLI parameters (-a and –q).

Our configuration file nvidia-settings-rc has added these lines relevant for frame lock settings:

0.0:0[gpu:0]/FrameLockMaster=65536
0.0:0[gpu:0]/FrameLockSlaves=131072
0.0:0[gpu:1]/FrameLockMaster=0
0.0:0[gpu:1]/FrameLockSlaves=196608
0.0:0[framelock:0]/FramelockUseHouseSync=0

As we use Xinerama and Twinview options in our X-Window configuration, we have two graphic cards, two screens and two monitors. In some cases kernel is not able to find and configure both graphics cards, so administrator needs to find correct busID in PC configuration (please note that in some tools these IDs are in hexadecimal format and xorg.conf needs decimal format). Relevant parts of our xorg.conf file follows:

Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0" 0 0
    Screen      1  "Screen1" 0 1080
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
EndSection

Section "ServerFlags"
    Option         "Xinerama" "1"
EndSection

Section "Monitor"
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "ATO DM-3400"
    HorizSync       13.0 - 68.0
    VertRefresh     23.0 - 61.0
    Option         "DPMS"
EndSection

Section "Monitor"
    Identifier     "Monitor1"
    VendorName     "Unknown"
    ModelName      "ATO DM-3400"
    HorizSync       13.0 - 68.0
    VertRefresh     23.0 - 61.0
EndSection

Section "Device"
    Identifier     "Videocard0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "Quadro FX 4600"
    BusID          "PCI:7:0:0"
EndSection

Section "Device"
    Identifier     "Videocard1"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "Quadro FX 4600"
    BusID          "PCI:134:0:0"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Videocard0"
    Monitor        "Monitor0"
    DefaultDepth    24
    Option         "TwinView" "1"
    Option "metamodes" "DFP-0:1920x1080+0+0,DFP-1:1920x1080+1920+0"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

Section "Screen"
    Identifier     "Screen1"
    Device         "Videocard1"
    Monitor        "Monitor1"
    DefaultDepth    24
    Option         "TwinView" "1"
    Option "metamodes" "DFP-0:1920x1080+1920+0,DFP-1:1920x1080+0+0"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

On top of this system SAGE version 2.5 is run with standard configuration for four displays.

Our fsmanager.conf file follows:

fsManager       cs 195.113.205.10
systemPort      20002
uiPort          20001
trackPort       20003
conManager      127.0.0.1 15557    

tileConfiguration stdtile-4.conf
receiverSyncPort  12000
receiverStreamPort   22000
receiverBufSize    100
fullScreen     1
winTime		0
winStep     20

rcvNwBufSize 16777216
sendNwBufSize 1048576 
MTU 8800

Our stdtile-4.conf file follows:
TileDisplay
	Dimensions 2 2 
	Mullions 0.625 0.625 0.625 0.625
	Resolution 1920 1200
	PPI 90
	Machines 1

DisplayNode 
	Name local
	IP 127.0.0.1
	Monitors 4 (0,0) (1,0) (0,1) (1,1)

There is no audio support in this SAGE system.

5  Post production devices

Main postproduction system we have access to, is FilmLight Baselight FOUR connected at CINEPOST office at Barrandov. Baselight FOUR is a closed Linux based system for color correction that uses cluster architecture for computation and storage of algorithms of picture modifications. It has four node computing cluster (that is fixed at Barrandov), shuttle host (moveable PC) and console (moveable special controller board).

In addition we run disk array Avid LanShare EX, on-line storage that allows real-time collaboration editing over networks for Avid editing software with about 6TB of capacity.

6  Conclusions

We have built a CineGrid node in Prague as open and distributed architecture that allows Czech and European research and creative communities to use their tools for long distance collaboration using high speed photonic networks.

These technologies also create opportunities to strengthen Czech and European film industry position on the global market.

6.1  Future developments

We plan to extend our CineGrid node to more localities in order to enable use of high-quality video and audio over photonic networks in more organisations. Academic, research and commercial instutions are welcome to join Prague MeidaNet and access shared devices.

We plan to extend functionality deploying CineGrid Exchange node, a storage system able to stream high-quality media and preserve CineGrid content.

References

[1]	HOLUB, P.; MATYSKA, L.; LIŠKA, M.; HEJTMÁNEK, L.; DENEMARK, J.; REBOK, T.; HUTANU, A.; PARUCHURI, R.; RADIL, J.; HLADKÁ, E.: High-definition multimedia for multiparty low-latency interactive communication. Future Generation Computer Systems, Amsterdam, The Netherlands. Elsevier Science, 2006, vol. 22, no. 8, pp. 856–861. ISSN 0167-739X.
[2]	HOLUB, P.: Updates on UltraGrid Platform. Internet2 Spring 2008 Member Meeting, Arlington, VA, USA, 2008. I2 Available online