so as solution we decided to encode audio with different bitrates (bitrate depends on receiver connection speed) seems might be ok, but its quite expensive to run 3 or more encoders in the same time even on desktop (and almost impossible on mobile platforms).. after couple weeks fighting with this problem i decided to go different way, the most expensive part of all MPEG audio encoders is Psycho acoustic  model, but it not much depends on destination bitrate… as result I did implement multi bitrate MPEG 1 Layer 2 audio encoder that can encode in up 10 different bitrates in the same time with cpu usage pretty similar to single bitrate encoder.

I was not surprised that it works pretty well on Desktop (Windows/Linux) and mobile platforms (Windows CE/Android)

this post is kinda announcement of new project that i start i`m sure that that kind of encoder will be pretty useful for audio conference software or for online radio stations for example

i was working on detecting microphone sound level. So i received buffer of 16 bit samples and to detect level i had to find max absolute value. The simplest way just go through buffer and found max value:

__int16 find_abs_max(__int16 * data, unsigned size)
{
__int16 max = 0;
for (unsigned i = 0; i < size; ++i)
{
if (abs(data[i]) > max)
max = abs(data[i]);
}
return max;
}

what can we do to increase performance? sure we can sort data and we will have max value, but sometimes especially when we have huge array of data fast sort algorithm wont help for example some sort algorithms require one more additional buffer with same size. if algorithm does not require additional buffer can happen that we cannot modify current data buffer as it should be passed somewhere as in my case…

So if you not satisfied performance you could try SIMD as I did.

__int16 find_abs_max_sse(void * data, unsigned size)
{
__declspec(align(16)) __int16 return_buf[8];
__int16 * dst = &return_buf[0];

__int32 _size = size >> 4;

__asm
{
mov ecx, _size
mov esi, data
mov edi, dst

pxor xmm0, xmm0
pxor xmm6, xmm6
pxor xmm7, xmm7

NEXT:
movupd xmm1, [esi]
pmaxsw xmm6, xmm1
pminsw xmm7, xmm1

add esi, 16
dec ecx
or ecx, ecx
jnz NEXT

psubw xmm0, xmm7
pmaxsw xmm6, xmm7

movdqa [edi], xmm6
};

__int16 max = 0;
for (unsigned char i = 0; i < 8; ++i)
{
if (return_buf[i] > max)
max = return_buf[i];
}

return max;
}

measurement showed that SSE implementation 20 times faster (!!!) if we dont need max by absolute value it should twice faster!

P.S. my SSE implementation work 100% correct only if buffer size in bytes dividable by 16 if its not true then we skip couple samples at the end, but its not difficult to fix.

Here is first version of HTCCamera class that works directly with driver as result we can choose resolution of grabbing frame, we can grab frames to memory without saving to storage. also it allows to control flash and autofocus!

In future it will allow to control brightness, saturation, hue, whiteballance so on…

First vesion you can download here

Ok you dont need anything special and any lib, for example you draw target with

Rect _rect

then to grab that part you need:

1. Grab full image

Bitmap bmp = new Bitmap(width, height);
System.Drawing.Imaging.BitmapData data = bmp.LockBits(new Rectangle(0, 0, width, height),
System.Drawing.Imaging.ImageLockMode.ReadWrite, System.Drawing.Imaging.PixelFormat.Format16bppRgb565);

bFlag = cam_.getRgb565(data.Scan0);
bmp.UnlockBits(data);

2. Create 2nd bitmap

Bitmap bmp2 = new Bitmap(_rect.Right - _rect.Left, _rect.Bottom - _rect.Top);

3. Using Graphics cut required part

gr.DrawImage(bmp, new Rectangle(0, 0, bmp2.Width, bmp2.Height),
new Rectangle(_rect.Left, _rect.Top, bmp2.Width, bmp2.Height), GraphicsUnit.Pixel);

bmp2 - result bitmap and its exactly what you need

[ComVisible(true), ComImport,
Guid("2B21644A-D405-4E27-A51C-A4812bE0CE4C"),
InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IGetFrame
{
[PreserveSig]
int getFrame(IntPtr pBuff);

[PreserveSig]
int getSize([Out] out long size);

[PreserveSig]
int getFrameParams(
[Out] out int width,
[Out] out int height,
[Out] out RawFrameFormat format);

[PreserveSig]
int drawText(
[In] IntPtr ptr,
[In] int height,
[In] int width);

[PreserveSig]
int stopDraw();

[PreserveSig]
int getGrayScale(IntPtr ptr);

[PreserveSig]
int getRgb(IntPtr ptr);

[PreserveSig]
int drawTarget(Rect rect, int type);

[PreserveSig]
int stopDrawTarget();
}

P.S. dont forget that rect depends of frame size, not control size.

Link to new library version

[DllImport("DirectShowNETCF.Native.dll")]
private static extern IntPtr GetBaseFilter(int index);

[DllImport("DirectShowNETCF.Native.dll")]
private static extern void DeleteBaseFilter(int index);

also interface IBaseFrame has been changed:

[ComVisible(true), ComImport,
Guid("2B21644A-D405-4E27-A51C-A4812bE0CE4C"),
InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IGetFrame
{
[PreserveSig]
int getFrame(IntPtr pBuff);

[PreserveSig]
int getSize([Out] out long size);

[PreserveSig]
int getFrameParams(
[Out] out int width,
[Out] out int height,
[Out] out RawFrameFormat format);

[PreserveSig]
int drawText(
[In] IntPtr ptr,
[In] int height,
[In] int width);

[PreserveSig]
int stopDraw();

[PreserveSig]
int getGrayScale(IntPtr ptr);

[PreserveSig]
int getRgb(IntPtr ptr);

[PreserveSig]
int drawTarget();

[PreserveSig]
int stopDrawTarget();
}

all this changes was made for new functionality (grabbing rgb or grayscale frames for RGB565 or YV12 fromats, drawing target on preview, using more then 1 sample grabber in 1 application)

You can check updated AMCamera example from DirectShowNETCF

I see there only 1 way to solve it - Draw text directly on stream, so we need SampleGrabber to got access to raw stream, also we need to create our own font (thats the main problem). i made small demo that allows draw text on preview.

So IGetFrame was modified:

[ComVisible(true), ComImport,
Guid("2B21644A-D405-4E27-A51C-A4812bE0CE4C"),
InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IGetFrame
{
[PreserveSig]
int getFrame(IntPtr pBuff);

[PreserveSig]
int getSize([Out] out long size);

[PreserveSig]
int getFrameParams(
[Out] out int width,
[Out] out int height,
[Out] out RawFrameFormat format);

[PreserveSig]
int drawText(
[In] IntPtr ptr,
[In] int height,
[In] int width);

[PreserveSig]
int stopDraw();
}

As you can see we got new method: drawText([In] IntPtr ptr, [In] int height, [In] int width);
where ptr - is our text in special format…
how to conver text to IntPtr?
there added extra class: CABC, here is example how to draw text:

CABC abc = new CABC("abcd dabc cdab bcda");
frmGrabber.drawText(abc.getText(), abc.Height, abc.Width);
abc.Dispose();
abc = null;

Also you can check AMCamera example from DirectShowNETCF

Why do we need SampleGrabber?
1. In windows mobile this is the fastest way to still images, more than that we still uncompressed images (so we dont loose time for encoding/decoding)
2. We can use SampleGrabber to draw text or images on received stream
3. We grab images to memory.

I Think this is enough to say that sample grabber is useful.
The problem is that windows already have sample grabber and we dont need to make any extra moves to use it, but Windows Mobile knows nothing about sample grabber

On www.codeproject.com you can find some articles that explain how to implement SampleGrabber. But there tons limits and problems in every case, so i decided to implement my own.

The main question was “Is it possible to implement SampleGrabber that not need to registered?”. I meant that if i programming on C++ i can use unit (class) that contain samplegrabber code, i if i programming on any other language i can use this unit compiled to dll. Yes its possible (!!!!) we dont need resvrCE and anything else, just unit or dll

I was going to use developed SampleGrabber for barcode reader, but samples that i receive were too blurred to be recognized even by human (may be this is my cam dieing? O,o). anyway that was reason i decided to stop work on it, so here is what i got for this time:

DirectShowNETCF.Native.dll that contain SampleGrabber. How to use it?

[DllImport("DirectShowNETCF.Native.dll")]
private static extern IntPtr GetBaseFilter();

this is method returns IntPtr that equal to IBaseFilter*

how to get IBaseFilter?

IntPtr grabber_ = GetBaseFilter();
IBaseFilter grabber = (IBaseFilter)Marshal.GetTypedObjectForIUnknown(grabber_, typeof(IBaseFilter));

To grab raw frame you have to call
1. getSize(out long size)
2. alloc memory (use DirectShowNETCF.PInvoke.LocalAlloc(0×40, size))
3. call getFrame(IntPtr pBuff)

[ComVisible(true), ComImport,
Guid("2B21644A-D405-4E27-A51C-A4812bE0CE4C"),
InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IGetFrame
{
[PreserveSig]
int getFrame(IntPtr pBuff);

[PreserveSig]
int getSize([Out] out long size);
}

//example of grabbing frame
public IntPtr grabFrame(ref long bufSize)
{
IntPtr res_ = IntPtr.Zero;
long size = 0;
frmGrabber.getSize(out size);
bufSize = size;
res_ = PInvoke.LocalAlloc(0x40, (int)size);
frmGrabber.getFrame(res_);
return res_;
}

last version of DirectShowNETCF you can download here

What do we need to change resolution?
1. We need Enum supported:

• Find STILL pin
• using IEnumMediatypes enumerate supported mediatypes

2. Using IAMStreamConfig set required mediatype to required pin (STILL pin in our case)

here is interfaces:

[ComVisible(true), ComImport,
Guid("89C31040-846B-11CE-97D3-00AA0055595A"),
InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IEnumMediaTypes
{
[PreserveSig]
int Next(
[In] int cMediaTypes,
[Out] out IntPtr ppMediaTypes,
[Out] out int pcFetched
);

[PreserveSig]
int Skip([In] int cMediaTypes);

[PreserveSig]
int Reset();

[PreserveSig]
int Clone([Out] out IEnumMediaTypes ppEnum);
}

[ComVisible(true), ComImport,
Guid("C6E13340-30AC-11D0-A18C-00A0C9118956"),
InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IAMStreamConfig
{
[PreserveSig]
int SetFormat([In, MarshalAs(UnmanagedType.LPStruct)] AMMediaType pmt);

[PreserveSig]
int GetFormat([Out] out IntPtr pmt);

[PreserveSig]
int GetNumberOfCapabilities(
[Out] out int piCount,
[Out] out int piSize);

[PreserveSig]
int GetStreamCaps(
[In] int iIndex,
[Out] out IntPtr ppmt,
[In] IntPtr pSCC
);
}

Also was created useful class CEnumMediatypes (as in DSPack)

public class CEnumMediaTypes
{
private List<AMMediaType> FList;
private IEnumMediaTypes EnumMediaTypes;
private IntPtr Mt;
public void Free()
{
Clear();
}

public CEnumMediaTypes()
{
FList = new List<AMMediaType>();
}

private void Clear()
{
for (int i = 0; i < FList.Count; i++)
{
DirectShowEx.FreeMediaType(FList[i]);
FList[0] = null;
FList.RemoveAt(0);
}

FList.Clear();
}

public int Count
{
get
{
return FList.Count;
}
}

public void Assign(ICaptureGraphBuilder2 capGraph, IBaseFilter filter, Guid category)
{
Clear();

IPin pin_ = null;
CGuid guid = new CGuid(category);
CGuid guid2 = new CGuid(CLSID_.MEDIATYPE_Video);
int res = capGraph.FindPin(filter, PinDirection.Output, guid, guid2, true, 0, out pin_);

if (res > -1)
{
pin_.EnumMediaTypes(out EnumMediaTypes);
if (EnumMediaTypes == null)
{
return;
}

int fetched = 0;

int hr = EnumMediaTypes.Next(1, out Mt, out fetched);
while (hr == 0)
{
fetched = 0;
hr = EnumMediaTypes.Next(1, out Mt, out fetced);
AMMediaType mt = (AMMediaType)Marshal.PtrToStructure(Mt, typeof(AMMediaType));
FList.Add(mt);
}

Marshal.ReleaseComObject(EnumMediaTypes);
Marshal.ReleaseComObject(pin_);
}
}

private string getFourCC(int value)
{
IntPtr ptr = Marshal.AllocHGlobal(4);
Marshal.WriteInt32(ptr, value);
byte[] bt = new byte[4];
Marshal.Copy(ptr, bt, 0, 4);
Marshal.FreeHGlobal(ptr);
char[] ch = new char[4];
for (int i = 0; i < 4; i++)
{
ch[i] = (char)bt[i];
}
bt = null;
return new string(ch);
}

public string GetMediaDescription(int index)
{
string result = "";

AMMediaType tempType = FList[index];
if (tempType.formatType == CLSID_.VideoInfo)
{
result += ((VideoInfoHeader)Marshal.PtrToStructure(tempType.formatPtr, typeof(VideoInfoHeader))).BmiHeader.Width.ToString();
result += 'X';
result += ((VideoInfoHeader)Marshal.PtrToStructure(tempType.formatPtr, typeof(VideoInfoHeader))).BmiHeader.Height.ToString();
}
else
{
if (tempType.formatType == CLSID_.VideoInfo2)
{
result += ((VideoInfoHeader2)Marshal.PtrToStructure(tempType.formatPtr, typeof(VideoInfoHeader2))).BmiHeader.Width.ToString();
result += 'X';
result += ((VideoInfoHeader2)Marshal.PtrToStructure(tempType.formatPtr, typeof(VideoInfoHeader2))).BmiHeader.Height.ToString();
}
}

tempType = null;
return result;
}

public AMMediaType this[int index]
{
get
{
return FList[index];
}
}

}

Exaple how to set/change resolution is here

So what do we need? tons extra com interfaces (Ipin, IEnumPins, IEnumFilters, IFileSinkFilter, IAMVideoControl and so on).

Also i implemented 2 useful classes (as for me) CPinList and CFilterList (the same as in DSPack).

public class CFilterList
{
private List<IBaseFilter> Filters;
private IEnumFilters enumFilters = null;
public CFilterList()
{
Filters = new List<IBaseFilter>();
}
~CFilterList()
{
Free();
Filters = null;
}
public void Assign(IGraphBuilder fg)
{
Clear();
fg.EnumFilters(out enumFilters);
IBaseFilter Filter;
int fetched = 0;

while (enumFilters.Next(1, out Filter, out fetched) == 0)
{
Filters.Add(Filter);
}

Marshal.ReleaseComObject(enumFilters);
enumFilters = null;
}
public int Count
{
get
{
return Filters.Count;
}
}
public IBaseFilter this[int index]
{
get
{
return Filters[index];
}
}
public void Free()
{
Clear();
}
private void Clear()
{
while (Filters.Count > 0)
{
Marshal.ReleaseComObject(Filters[0]);
Filters[0] = null;
Filters.RemoveAt(0);
}

Filters.Clear();
}
}

public class CPinList
{
private List<IPin> Pins = null;
private IEnumPins enumPins = null;

public CPinList()
{
Pins = new List<IPin>();
}

~CPinList()
{
Free();
Pins = null;
}

public int Count
{
get
{
return Pins.Count;
}
}

public IPin this[int index]
{
get
{
return Pins[index];
}
}

public void Assign(IBaseFilter filter)
{
Clear();
filter.EnumPins(out enumPins);

IPin pin;
int fetched = 0;

while (enumPins.Next(1, out pin, out fetched) == 0)
{
Pins.Add(pin);
}

Marshal.ReleaseComObject(enumPins);
enumPins = null;

}

private void Clear()
{
while (Pins.Count > 0)
{
Marshal.ReleaseComObject(Pins[0]);
Pins[0] = null;
Pins.RemoveAt(0);
}
Pins.Clear();
}

public void Free()
{
Clear();
}
}

so now stilling images works, btw in the same time with preview, but there some bad news (if we can call it so):
a) it takes like 15 seconds to build graph
b) when i still image preview hangs for like 2 seconds.

good news:
a) we can implement samplegrabber filter and do not wait when stilling images and building graph
b) filter of video capture device has 3 output pins (on my device), i`m not sure, but seems like: preview, still image and probably still video will try to check it soon

And last one perhaps preview and stilling images wont work on some devices in the same time so you will need to choose what exactly you need (but this is only unconfirmed “perhaps”)

Almost forgot