Microserver Based Surveillance System Combines Scalable Ceph Storage And Nx Server VMS system16 Jul, 2020
White Paper - A scalable & High availability Arm Microserver based Converged Video Management System
Server-based Surveillance System Introduction
The components of a digital surveillance system include IP cameras, storage systems, and video management software to record and manage the live and recorded video. There are generally two types of surveillance systems; the Embedded system based NVR or server-based VMS. Compare to the embedded system NVR, server-based VMS provides much flexibility, scalability, and intelligence. A larger-scale surveillance system will usually choose the server-based VMS solution.
Server-based VMS recording software can run on Linux or Windows-based servers that have an internal disk array for recorded video. Or, VMS servers do recording jobs only and store video to an external storage system. The secondary solution offers better flexibility because of the scales of the recording server and storage server are independent.
If you have a new requirement for adding more cameras or storage capacity, you can add servers for the new requirement.
Pain Points Require Improvements
However, there are still some pain points that exist in server-based VMS that need to be addressed.
- Rebuilding a failed disk in disk array takes too much time. The risk of lost data increases with the size of the disk. No matter the VMS servers use an internal storage or external storage system, they use the RAID controller to protect data from loss. If a disk drive fails, the disk array controller will take a very long time to rebuild the files to the spare drive. The rebuild time and the possibility of failure to rebuild drive are proportional to the size of the disk. Today, the largest hard drive is 16TB and sooner 20TB drive will be available. Using small size disks increases the total cost of storage servers.
- The storage throughput suffers a big impact during the degrade rebuilding mode.
- The traditional server has a large failure domain. If a server fails, all videos inside the server become inaccessible or even lost. More disks are installed in a server, more video files are lost at the same time when the server fails.
- VMS lack for an elastic hardware architecture.
Use Software-Defined Storage and Arm MicroServer with Network Optics Nx Witness Video Management System
In this article, we will present how Ambedded's software-defined storage appliance and Arm-based microserver integrate Network Optics' VMS software to solve some of the issues that exist in the current solution.
Software-Defined Storage Appliance
Ambedded's software-defined storage Mars 400 Ceph Storage Appliance is a turn-key software-defined storage solution for enterprises. The appliance integrates ARM-Based microserver with storage software Ceph and web user interface. Mars 400 provides a unified storage cluster that supports the object, block, and file system. It is designed to provide excellent performance, reliability, and scalability with no single point of failure and minimum failure domain.
Nx Witness Video Management System & Nx Server
Nx Witness VMS is an open IP video management system(IPVMS) designed to find, view, record, and manage IP video cameras as well as integrate quickly with 3rd party systems and devices. The Nx Server application is a lightweight, powerful media server responsible for discovering, and managing Nx Witness system devices and data. Nx Server records video from IP cameras to inside or externally attached storage. It has server hive architecture ensuring no single point of failure.
Arm Microserver Architecture
The microserver platforms accommodate 8x Quad-Core ARM 64-bit servers modules in a one-rack-unit (1RU) chassis. Two in-chassis redundant switches provide 4x10 Gbps uplink for horizontal scale-out and client access. Every ARM microserver module is an independent Linux server that owns dedicated CPU, memory, storage, and network interface resources for application software.
The microserver architecture provides greater accessibility due to its lack of a single point of failure. Every microserver node can be configured as a storage node of a storage cluster or a VMS server node of the VMS cluster.
When the microserver is configured as a storage node, it joins the Ceph storage cluster to work with other storage nodes. If we configure the microserver as an Nx Server, it does his job as a VMS server and mounts a virtual disk from the Storage cluster for storing the recorded video files. This means that you can flexibly deploy the necessary number of Nx servers and Ceph storage servers all on Mars 400 microservers. Because of the advantages of microserver, you have the flexibility of scale the surveillance system with a small increment.
How Software-Defined Storage Protect and Reheal Data Against Hardware Failures
Ceph storage uses replication or erasure code algorithm to protect data from loss instead of the RAID controller. Erasure coding is analog to the RAID 5 or RAID 6 in the disk array but with much flexibility of allowing more disk fail currently. It can protect data not only from disk drive failures but also protect data against server failures, rack failure, and any scale of failure domain. Data or video files are first stripped to objects and every object is split to a number of data chunks and calculated redundant chunks. The storage server distributedly stores data and redundant chunks to the different storage servers in the storage cluster. If a storage server breaks, all data stored inside that server have to be rehealed. Data healing is done by all healthy servers and disks. And, only the lost chunks have to be re-healed instead of a rebuild all of the whole disks.
There are three major advantages of distributed software-defined storage compare to the RAID storage system.
- Data can survive from one or more server node breaks.
- Data reheal is much faster than RAID because all healthy disks in the cluster help data healing in parallel and only lost data need healing. If there are more disks in the cluster, healing speed is faster. An experiment of losing one 60% usage 10TB drive in a 36 drive cluster took 2.5 days for reheal. The reheal speed is 28.5MB/s for this experiment scale. (Clay Erasure code 4+2 profile)
- During the reheal, healing does not interrupt the video recording
- You can use large capacity disk drives such as 16TB HDD in the Ceph storage cluster to get the same capacity with fewer servers.
How Many Camera Recording can One Microserver Support
We set up a testing environment to stress the Nx Server on Ambedded Mars 400 microserver. The experiment firstly used a server to simulate IP cameras for streaming videos to an Arm microserver. The video stream format is H.264, 1080p high quality at 30 frames per second. The Arm VMS server mounts a virtual disk as its storage. The test result concludes that one microserver can record a maximum of 30 cameras without losing the video frame. The testing also includes the scenario while the virtual disk is full. This implies that a 1U Mars 400 that accommodates eight microservers can support 240 cameras high definition videos. This implies that a 1U Mars 400 that accommodates eight microservers can support up to 240 camera high definition video streams. This is a great achievement by using a 1U server and consumes only 100 Watts of power.
Test with Real IP Cameras
In another test, we use fifty IP cameras for making the test in a real environment.
- IP Cameras: 25x Wired and 25x Wireless
- Mars 400Nx server: 8 nodes in 1U. 4 cores Arm A72, 4GB RAM & 2x 2.5Gbps LAN
- Nx Server: 3 Mars 400Nx Arm nodes,
♦ 3 nodes are active for recording and playback. There is a spare resource on each node for one node failover.
♦ Data protection: erasure code 4+2, allow up to 2 hard drive fails at the same time
♦ Ceph server: 21x microserver for Ceph OSD + 3x microserver for Ceph monitor
♦ Hard drives: 21x 16TB Seagate Exos Enterprise SATA HDD. 224TB is usable.
♦ 4x 10Gb ports for 4x Mars 400Nx
♦ 1x Gb switch for cameras
♦ Wireless access point x 2
During recording and replay, only 150% of CPU resource is used. The maximum usable CPU resource is 400%. Please refer to figure 1.
Figure 1: Nx Server node CPU usage
There is a total of 4GB memory available in each node. 1GB memory is reserved for spare use. The memory amount used during the test is about 2.8 GB. The server node can stably support up to 30 cameras.
Figure 2: Nx Server node unused memory
The network traffic during the test is about 150Mbps download and 130Mbps for upload. The 2x 2.5Gb network bandwidth is much higher than the real traffic.
Figure 3: Nx Server nodes network traffic
VMS Server Node FailOver
Network Optics VMS server has the automatic server failover feature, which allows healthy VMS servers to connect cameras from a failed Nx VMS server. In our test, we set up three microservers as the Nx VMS server. They can handle a total of 90 cameras. In the normal condition, 50 cameras under test are connected evenly to the three cameras. We simulate a server offline by turning it off. We prove 18 cameras connected to the failed server can automatically reconnect to the other two servers in about one minute.
This implies that using the microserver solution for the Nx VMS server; you need only 1/8 of a 1U Mars 400Nx server spared for the failover. Compare to a big node server; microserver architecture can save one server cost for the failover setup.
- Number of cameras: 200 cameras
- Bit Rate per camera for 1080p, high quality, 30 FPS: 4.87Mbps
- Recording hours per day: 16 hours
- Days to record: 60 days
Calculation of server requirements
- The total usable storage capacity required: 411 TB
- The total raw capacity required if we use erasure code 4+2: 678 TB
- Hard drive capacity: 16 TB
- Number of hard disks required: 43 HDDs
- Number of node required for storage (Ceph OSD): 43
- Ceph Monitor node: 3
- Nx server nodes for 200 cameras: 200/30 = 7 nodes.
- Spare node for NxServer failover: 1 node
- Total microserver nodes required: 43 + 3 + 7 + 1 = 54 nodes
- Total Mars 400Nx required: 54/8 = 7x 1U Mars 40Nx
- Spare nodes: (7 x 8) - 54 = 2 nodes
- Total power consumption of server exclude HDD: 7 x 100 Watts = 700 Watts
Ambedded Mars 400Nx combines Nx VMS server, and software-defined storage has the following advantages:
- Mars 400Nx's Software-defined storage can protect data against not only disk fails, but also the server fails.
- When there are disk fails, Mars 400Nx re-healed data parallelly by all healthy drives. It is 4 to 5 times faster compared to the RAID controller.
- Mars 400Nx can maintain its video recording and replay during data healing.
- You can use large capacity disk drives such in the Mars 400Nx storage cluster to reduce the total cost of the storage system.
- A 1U Mars 400Nx VMS server can support up to 240x 1080p cameras. You can design the VMS system with a flexible combination of storage and VMS servers.
- Mars 400Nx Arm server saves 70 percent of power consumption compares to x86 servers.
Ambedded is a software-defined storage company with expertise on Linux OS, kernel, software-defined storage, embedded system, and Arm server.
Ambedded's vision is to make the use of Ceph software-defined storage easy so enterprises can benefit from Ceph with a shorter learning curve.
Ambedded makes Ceph simple by offering easy to use user interface, ready to use storage appliance, stability & performance tuning, automate deployment, and Professional support.
With the purpose-built Ceph appliance, the Ambedded team owns wide experience to help customers to adopt our ceph solution into versatile industries, such as Telecom, medical, military, edge data center, and enterprises who require high availability storage.
Visit the Ambedded web site for details.
About Network Optics
Network Optix is a software development company focused on creating cutting edge video management solutions. Our core product - Nx VMS - is an open, lightweight, customizable video management platform capable of managing offline media, live video streams, and integrated 3rd party systems and devices. Nx VMS is a free live-streaming platform - cloud enabled, remotely accessible, scalable to any size, and licensed only when recording and search of IP Video cameras are required. Nx VMS is also open for developers, with comprehensive API and SDK documentation built-in and free to use, allowing organizations or individuals to create their own video-driven applications for any vertical market or project.
We would like to thank Seagate and Network Optix for providing the Exos X16 Enterprise Hard Drives and IP cameras used for this project.
- Why RAID stop working in 2019, By Robin Harris for Storage Bits
- Why the best RAID configuration is no RAID configuration by Guest Blogger → https://blog.shi.com/hardware/best-raid-configuration-no-raid-configuration/
Mars 400Nx offers a high availability ceph storage to Nx VMS software
Combine Ceph Storage and Video Management System to be a converged digital surveillance system.
- Related Technolgy
- Related Products
Mars 400 Mars 400SES Datasheet
Ceph appliance & SUSE Enterprise storage datasheet
Ceph Storage Solutions Design and Manufacture | Ambedded
Located in Taiwan, Ambedded Technology Co., LTD. is a Ceph storage solution provider that offers ARM based software defined storage appliances with 60% CapEX saving, 70% less power consumption and 100% scale out advantages.
For Software-defined storage designing and manufacturing, Ambedded team has broad experience to help customers to adopt the ceph solution into versatile industries, such as Telecom, medical, military, edge datacenter and HA required enterprise storage.
Ambedded has been offering customers Ceph storage solutions since 2013, both advanced technology and 20 years of experience, Ambedded ensures each customer's demands are met.