Introduction to Analytics integration

You can integrate video analytics engines with XProtect in a number of ways.

Development considerations

Here, focus is on Video Content Analysis (VCA) engines, but you can integrate any analytics engine with XProtect. Two common ways of doing this are illustrated below in the Data Flow section.

You can take the following aspects into considerations:

Retrieving video from XProtect
Sending analytic events to XProtect
Developing a Plug-in for the Event Server
Developing a Plug-in for the Smart Client

In the following sections, the above areas are covered in detail.

Data flow

This diagram illustrates a data flow when the analytics engine is located on camera.

This diagram illustrates a data flow when the analytics engine is server based.

The overall data flow of the framework is as follows: the underlying analytic system detects a predefined behavior which is configured by setting up rules on the third-party VCA system, and then sends out an analytics event to the XProtect Event Server. The analytic system sends the event by using one of the protocols and formats described in this document. The address of the XProtect Event Server should be set up on the third-party VCA. The XProtect Event Server then matches the event message against a predefined alarm definition, and—if there is a match—an alarm is triggered. This alarm appears in the XProtect Smart Client alongside any other alarms generated by the XProtect Event Server, and is managed in exactly the same way.

Screenshot from XProtect Smart Client: The effects of analytics events in the XProtect Smart Client with loaded data in the alarm list.

Retrieving video from XProtect

To avoid developing protocol support for all the different camera models and video servers, you can retrieve video from XProtect. You can retrieve video both in live mode, with only a few milliseconds delay through the XProtect server, and in playback mode when the video has been recorded.

The following methods are available for retrieving video:

ImageServer Protocol
1. This provides the video as it was sent from the camera with a small Milestone header (Generic Byte Data).
2. In live mode the frames are received frame by frame, in playback mode the full GOPs are retrieved (for MPEG-4, H.264 and H.265).
3. The codec can be any of the supported codecs.
Media Toolkit
1. This component provides access to decoded video.
2. Several formats are supported, for example BGR and RGB in 24 or 32 bits pixel depth, and JPEG.
3. Grey tone and resizing can also be requested.
4. Cropping can be requested.
5. Can access live and playback video from XProtect Recording Servers, from exported databases, and from XProtect databases.

When you execute the Analytics engine on a Windows platform, Milestone recommends that you use the Media Toolkit.

Sending analytics events to XProtect

The decision of how to send an analytics event to the XProtect VMS has impact on how the event can be used for alarm configuration and how it is displayed to the operators.

The following methods are available when you want to notify XProtect of an analytics event:

Send a simple event, no extra information.
Send an analytics event (MAD) including detailed information
Send an alarm, including detailed information

Simple event

You can implement the simple event by using a user-defined event, where the sender needs to know the guid of a configured user-defined event. Or you can use the Generic protocol to send text to be interpreted.

For smaller systems, this approach works well, but for larger systems, this approach requires a lot of configuration.

Analytics events (MAD)

When you want to include detailed information, but the executing environment is not Windows or .Net cannot be used, then you can send a MAD formatted event to the Event Server or to the XProtect Analytics server at default port 9090.

The MAD XML has a node 'Rule' that is used by the XProtect Event Server to identify matching alarms, the content of the 'Rule' node is compared to the message text in Alarm Definitions. You can add new message texts by:

Making end users configure new message strings in the administrator (Analytics Events), or
Creating an XProtect Event Server plug-in that defines the set of new messages ( ItemManager.GetKnownEventTypes ).

MAD XML sample:


<?xml version="1.0"  encoding="utf-8"?>
<Alert xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"   
       xmlns:xsd="http://www.w3.org/2001/XMLSchema"
       ID="0"
       TimeStamp="2008-09-29T20:06:02.0720778+02:00"
       Name="WalkingWrongWay"
       xmlns="http://tempuri.org/Alert.xsd">
  <Source>
    <Server GUID="CCCF8C51-94DB-49b2-9718-0800A151D76B"
            Name="10.10.11.26">
      <Description>Milestone Surveillance Server</Description>
    </Server>
    <Device GUID="D23414E5-B03F-4C78-8AB9-A36D1488F9C2"
            Hostname="10.10.50.28"
            Name="[Axis 221] Camera 1">
      <Location Name="Exit Door" />
    </Device>
  </Source>
  <Rule Name="WalkingWrongWay" Type="Behaviour">
    <Polygon>
      <Point X="241" Y="63" />
      <Point X="65" Y="140" />
      <Point X="75" Y="284" />
      <Color R="0" G="0" B="255" A="255" />
    </Polygon>
  </Rule>
  <Object ID="42" Name="John Doe" Type="Person" Size="2.39">
    <BoundingBox Top="194" Left="286" Bottom="429" Right="408">
      <Color R="255" G="0" B="0" A="255" />
    </BoundingBox>
    <Motion Speed="1.816466" ScreenSpeed="0.2389245" SpeedAngle="1.546658">
      <Path>
        <Point X="88" Y="288" />
        <Point X="129" Y="300" />
        <Point X="150" Y="304" />
        <Color R="0" G="255" B="0" A="255" />
      </Path>
    </Motion>
  </Object>
  <Description>Walking the wrong way at the exit door</Description>
  <Vendor Name="VCA Vendor" />
</Alert>

Analytics events (MIP .NET library)

When you execute in Windows and you are able to use .NET, then you can use the .NET library for MIP to more easily create events and alarms.

When sending new events to the XProtect Event Server, the content of the message text needs to be known by the XProtect Event Server for an Alarm Definition can be configured. You can add new message texts by:

Making end users configure new message strings in the administrator (Analytics Events), or
Creating an XProtect Event Server plug-in that defines the set of new messages ( ItemManager.GetKnownEventTypes ).

Analytics events (Protocol)

You can use the AlarmCommand Web service to send analytics events and alarms to the XProtect Event Server. The Proxy can be generated from an installed system, or can be instantiated from the MIP .NET library. This can be done when you execute your application on a Windows platform.

Developing a plug-in for the XProtect Event Server

It is not a requirement to develop a plug-in for analytics integration. You can use a plug-in for the XProtect Event Server:

When analytics events are already being sent from VCA, but in a different protocol format. An Event Server plug-in could retrieve this format and convert it to Milestone format.
When the analytics events need to be correlated with other states or events, before turning to alarms.
When the analytics event devices/servers need to be available on a map in the XProtect Smart Client, for example for checking operational status or for allowing context menu commands.
To provide known analytics rule and messages for easier configuration by end-user.

Developing a plug-in for the XProtect Smart Client

It is not a requirement to develop a plug-in for the XProtect Smart Client. However, a plug-in in the XProtect Smart Client can provide additional features for developing a specific display of the analytics event content, for example for interpreting the vendor specific information.