Blog

Image Manipulation in Windows CE or Mobile using VS .NET

Image Manipulation in Windows CE or Mobile using VS .NET

Recently DMC developed a high speed vision inspection system using LabVIEW. This system inspects many different parts and each part requires different inspection criteria, so we needed to develop a user interface (UI) that could be used to configure the current products, as well as be flexible enough to configure new products.


The manufacturing machine was already running on a Windows CE device, so our UI would need to interface with the UI already on the system and run on Windows CE. For a faster development time we decided to go with C# .NET for Windows CE. Unfortunately, the .NET library and functionality for Windows CE is far less capable than the standard .NET library.


The Windows CE library was sufficient for most of our needs, but there were two areas that proved fairly difficult. The first difficult item was obtaining and decoding the custom TCP messages that contained the images and inspection results from the LabVIEW Embedded Vision System that was running the vision inspection. However, after spending some time encoding and decoding the images, the UI was able to access full-sized images from the inspection system.
The next difficulty was that once these images were on the UI we needed to be able to zoom, pan, and draw inspection areas on the image. In a full .NET library this would have been really simple, but for a Windows CE device it required some custom work. The first step was to create a custom control and place just a picture box inside. Additionally, I set the SizeMode to StretchImage. By having the image take up whatever size the picture box was, I was able to zoom by increasing/decreasing the size of the picture box. Additionally, I was able to pan by moving the picture box relative to the custom control.


I have included all of the back-end code that I used to make the image behave as I wanted. The code I had was significantly more complicated, as I needed to be able to draw and pan rectangles that were used for inspection zones. I tried to remove all of that code to make the items I am explaining more clear; however, if anything seems odd, it is likely a remaining piece of the old code. If anyone is interested in the rectangle drawing code I would be happy to provide more information. Here is the simplified code:
 

using System;
using System.Linq;
using System.Collections.Generic;
using System.ComponentModel;
using System.Drawing;
using System.Data;
using System.Text;
using System.Windows.Forms;
using GlueInspectionHMI.Classes;
using DMCLabVIEWInterface;
using System.Diagnostics;

namespace GlueInspectionHMI.Controls
{
    public partial class ImageDisplay : UserControl
    {
        #region Fields
        //Private variables for size of image
        private Size m_oOriginalSize;  //Original Image Size
        private Size m_oSizeOfContainingBox;  //Size on control contianing pictureBox
        private Size m_oCurrentSize;  //Current Image size

        //Current Zoom Percentage
        private int m_iPercentZoom;
        //Zoom percantage that shows the entire image
        private int m_iFillZoom;

        //Variables used to store Pan Start
        private int m_iPanXStart, m_iPanYStart;

        //Total pan since last draw
        //Used to do a redraw after a certain pixels have been panned
        private int m_iTotalPanXSinceRedraw, m_iTotalPanYSinceRedraw;

        //Current Cursor Positions
        private int iCursorX, iCursorY;

        //Flag that sets on mouse down and resets on mouse up
        private bool m_bPanningImage = false;

        //Bool that denotes that image is a display image only
        private bool m_bFalseImage = false;

        //This is the current zoom percentage.  Make this read only
        public int iPercentZoom
        {
            get { return this.m_iPercentZoom; }
        }
        #endregion

        #region Constructor and Initialization functions
        //Default Contsructor
        public ImageDisplay ()
        {
            InitializeComponent();

            //Initialize the picturebox with a small bitmap off of the screen.
            this.pictureBox.Size = new Size( 1, 1 );
            this.pictureBox.Location = new Point( -1, -1 );
            this.pictureBox.Image = (Image)new Bitmap( 1, 1 );
        }

        //Contstructor with an image
        public ImageDisplay ( Bitmap newImage, Size oSizeOfContainingBox )
        {
            InitializeComponent();

            //Set the image
            this.pictureBox.Image = newImage;
            //Put image at top left location
            this.pictureBox.Location = new Point( 0, 0 );
            this.pictureBox.Size = newImage.Size;

            m_oOriginalSize = newImage.Size;
            m_iPercentZoom = 100;
            m_oSizeOfContainingBox = oSizeOfContainingBox;
            m_bFalseImage = false;
        }

        //Used to assign a new image to the display
        public void SetImage ( Bitmap newImage, Size oSizeOfContainingBox )
        {
            SetImage( newImage, oSizeOfContainingBox, false );
        }

        //Used to assign a new image to the display
        //Used when image should not be allowed to pan
        public void SetImage ( Bitmap newImage, Size oSizeOfContainingBox, bool bDisplayImageOnly )
        {
            m_oOriginalSize = newImage.Size;
            m_iPercentZoom = 100;
            pictureBox.Image = newImage;
            //Put image at top left location
            pictureBox.Location = new Point( 0, 0 );
            m_oSizeOfContainingBox = oSizeOfContainingBox;
            m_bFalseImage = bDisplayImageOnly;
        }
        #endregion

        #region Resizing and zoom
        //Resizes the image retaining aspect ratio.
        public void ResizePictureBox ( Size newSize )
        {
            pictureBox.Size = GetNewSize( newSize, m_oOriginalSize, out m_iPercentZoom );
            m_iFillZoom = m_iPercentZoom;

            //Of the width of the picturebox is less than the desired size then center the image
            if ( pictureBox.Size.Width < newSize.Width )
            {
                //Find number of pixels on either side and put the image in the center
                int iNumPixelsOnBothSides = (int)Math.Round( (double)( ( m_oSizeOfContainingBox.Width - pictureBox.Size.Width ) / 2 ) );
                this.pictureBox.Location = new Point( iNumPixelsOnBothSides, 0 );
            }
            //Else left justify    
            else
            {
                this.pictureBox.Location = new Point( 0, 0 );
            }

            //Resize any configured rectangles 
            for ( int i = 0; i < oArrayOfRectangles.Count; i++ )
            {
                oArrayOfRectangles[ i ].ScaleRectangle( m_iPercentZoom );
            }
        }

        //iPercentDelta is amount to zoom in or out by
        public void Zoom ( int iPercentDelta )
        {

            m_iPercentZoom = m_iPercentZoom + iPercentDelta;
            Size newSize = GetNewSize( m_iPercentZoom, m_oOriginalSize );

            //Pan so zoom is on center
            pictureBox.Size = newSize;
            SnapToEdge();

            for ( int i = 0; i < oArrayOfRectangles.Count; i++ )
            {
                oArrayOfRectangles[ i ].ScaleRectangle( m_iPercentZoom );
            }
            //Change size of selected rectangle
            oSelectedRectangle.ScaleRectangle( m_iPercentZoom );
        }

        //Zooms to an absolute zoom level
        internal void ZoomAbs ( int iZoomAmount )
        {
            Zoom( ( m_iFillZoom + iZoomAmount ) - m_iPercentZoom );
        }

        //Gets a new size while maintaining aspect ratio.
        private Size GetNewSize ( Size newSize, Size originalSize, out int iPercentZoom )
        {
            float nPercent = 0;
            float nPercentW = 0;
            float nPercentH = 0;

            nPercentW = ( (float)newSize.Width / (float)originalSize.Width );
            nPercentH = ( (float)newSize.Height / (float)originalSize.Height );

            if ( nPercentH < nPercentW )
                nPercent = nPercentH;
            else
                nPercent = nPercentW;

            iPercentZoom = (int)Math.Floor( nPercent * 100 );
            nPercent = (float)m_iPercentZoom / 100;
            int destWidth = (int)Math.Ceiling( ( (double)( (float)originalSize.Width * nPercent ) ) );
            int destHeight = (int)Math.Ceiling( ( (double)( (float)originalSize.Height * nPercent ) ) );

            return ( new Size( destWidth, destHeight ) );
        }

        //Gets a new size while maintaining aspect ratio.
        private Size GetNewSize ( int iPercent, Size originalSize )
        {
            if ( iPercent < m_iFillZoom )
            {
                iPercent = m_iFillZoom;
            }
            float nPercent = (float)iPercent / 100;
            int destWidth = (int)Math.Ceiling( (float)originalSize.Width * nPercent );
            int destHeight = (int)Math.Ceiling( (float)originalSize.Height * nPercent );
            m_iPercentZoom = iPercent;

            return ( new Size( destWidth, destHeight ) );
        }
        #endregion

        #region Panning
        //Pan Vertical function
        public void PanImageVertical ( int numPixels )
        {
            PanImageBothDir( 0, numPixels );
        }

        //Pan Horizontal Function
        public void PanImageHorizontal ( int numPixels )
        {
            PanImageBothDir( numPixels, 0 );
        }

        //Function to pan in both directions
        private void PanImageBothDir ( int iHorizontalPan, int iVerticalPan )
        {
            int xPosition;
            int yPosition;

            //If the image is smaller than the bounding box then center
            if ( pictureBox.Width < this.Width )
            {
                //Find number of pixels on either side and put the image in the center
                int iNumPixelsOnBothSides = (int)Math.Round( (double)( ( m_oSizeOfContainingBox.Width - pictureBox.Size.Width ) / 2 ) );
                xPosition = iNumPixelsOnBothSides;
            }
            //If the move will move the left edge past 0
            else if ( pictureBox.Location.X + iHorizontalPan > 0 )
            {
                xPosition = 0;
            }
            //If move will move the right 
            else if ( pictureBox.Location.X + iHorizontalPan + pictureBox.Width < this.Width )
            {
                xPosition = this.Width - this.pictureBox.Size.Width;
            }
            //Do standard move
            else
            {
                xPosition = pictureBox.Location.X + iHorizontalPan;
            }


            //If the image is smaller than the bounding box then top justify or if the new move will move the top edge past 0
            if ( pictureBox.Height < this.Height || pictureBox.Location.Y + iVerticalPan > 0 )
            {
                yPosition = 0;
            }
            //If move will move the bottom past the size of the box
            else if ( pictureBox.Location.Y + iVerticalPan + pictureBox.Height < this.Height )
            {
                yPosition = this.Height - pictureBox.Height;
            }
            //Do the move
            else
            {
                yPosition = pictureBox.Location.Y + iVerticalPan;
            }

            pictureBox.Location = new Point( xPosition, yPosition );
        }
        #endregion

        #region General Functions
        //This function prevents the image from showing the edge.  Will only allow images to go to the edge of the Control
        private void SnapToEdge ()
        {
            //If the box is out of the bounds on both ends leave it in place
            if ( pictureBox.Location.X <= 0 && ( pictureBox.Location.X + pictureBox.Width ) >= this.Width )
            {
                //Do nothing no need to snap
            }
            //If the image is smaller than the bounding box then center it
            else if ( pictureBox.Width < this.Width )
            {
                //Find number of pixels on either side and put the image in the center
                int iNumPixelsOnBothSides = (int)Math.Round( (double)( ( this.Width - pictureBox.Size.Width ) / 2 ) );
                this.pictureBox.Location = new Point( iNumPixelsOnBothSides, pictureBox.Location.Y );
            }
            //If there is space on left and image coverd up right then move to see most amount of image
            else if ( pictureBox.Location.X > 0 )
            {
                pictureBox.Location = new Point( 0, pictureBox.Location.Y );
            }
            //Otherwise there is space on right and image coverd up on left so put it as far right as possible
            else
            {
                pictureBox.Location = new Point( this.Width - pictureBox.Width, pictureBox.Location.Y );
            }

            if ( pictureBox.Location.Y <= 0 && ( pictureBox.Location.Y + pictureBox.Height ) >= this.Height )
            {
                //Do nothing no need to snap
            }
            else if ( pictureBox.Location.Y > 0 || ( pictureBox.Location.Y == 0 && ( pictureBox.Location.Y + pictureBox.Height ) < this.Width ) )
            {
                pictureBox.Location = new Point( pictureBox.Location.X, 0 );
            }
            else
            {
                pictureBox.Location = new Point( pictureBox.Location.X, ( this.Height - pictureBox.Height ) );
            }
        }

        //Internal function to determine if there is an image.  
        //When created has a fake image that is size 1,1, so will
        //return false if provided an actual image that is 1x1
        internal bool HasImage ()
        {
            if ( this.pictureBox.Image.Size == new Size( 1, 1 ) )
            {
                return false;
            }
            else if ( m_bFalseImage )
            {
                return false;
            }
            else
            {
                return true;
            }

        }

        //This function clears the image and removes it from memory
        //Used due to memory constaints on Windows CE device
        public void DisposeImage ()
        {
            if ( this.pictureBox.Image != null )
            {
                this.pictureBox.Size = new Size( 1, 1 );
                this.pictureBox.Location = new Point( -1, -1 );
                this.pictureBox.Image.Dispose();
                this.pictureBox.Image = (Image)new Bitmap( 1, 1 );
            }

            this.bDisplayRectangles = false;
        }
        #endregion

        #region Events
        //Event for Mouse Down (used to Pan)
        private void pictureBox_MouseDown ( object sender, MouseEventArgs e )
        {
            m_iPanXStart = e.X;
            m_iPanYStart = e.Y;
            m_iTotalPanXSinceRedraw = 0;
            m_iTotalPanYSinceRedraw = 0;
            m_bPanningImage = true;
            Debug.WriteLine( "Mouse Down --- X: " + m_iPanXStart.ToString() + "  Y: " + m_iPanYStart.ToString() );
        }

        private void pictureBox_MouseMove ( object sender, MouseEventArgs e )
        {
            //Used for panning if mouse is down
            iCursorX = e.X;
            iCursorY = e.Y;
            if ( e.Button == MouseButtons.Left )
            {
                if ( m_bPanningImage )
                {
                    PanImageBothDir( e.X - m_iPanXStart, e.Y - m_iPanYStart );
                }
            }
        }

        //Mouse Up Event
        private void pictureBox_MouseUp ( object sender, MouseEventArgs e )
        {
            if ( e.Button == MouseButtons.Left )
            {
                m_bPanningImage = false;
            }
        }

        //Function to resize picture box if a resize happens in the main UI
        private void pictureBox_Resize ( object sender, EventArgs e )
        {
            m_oCurrentSize = pictureBox.Size;
        }

        #endregion

        #region Public functions to modify the selected rectangle
        public bool SetBaseRectXForSelectedRect ( int iXPos )
        {
            int iIndexOfSelectedRect = oArrayOfRectangles.IndexOf( oSelectedRectangle );
            if ( iIndexOfSelectedRect == -1 )
            {
                return false;
            }
            else
            {
                oArrayOfRectangles[ iIndexOfSelectedRect ].SetBaseRectX( iXPos, m_iPercentZoom );
                oSelectedRectangle = oArrayOfRectangles[ iIndexOfSelectedRect ];
            }
            this.pictureBox.Invalidate();
            OnRaiseSelectedRectangleChangedEvent( new ROI( oSelectedRectangle ) );
            return true;
        }

        public bool SetBaseRectYForSelectedRect ( int iYPos )
        {
            int iIndexOfSelectedRect = oArrayOfRectangles.IndexOf( oSelectedRectangle );
            if ( iIndexOfSelectedRect == -1 )
            {
                return false;
            }
            else
            {
                oArrayOfRectangles[ iIndexOfSelectedRect ].SetBaseRectY( iYPos, m_iPercentZoom );
                oSelectedRectangle = oArrayOfRectangles[ iIndexOfSelectedRect ];
            }
            this.pictureBox.Invalidate();
            OnRaiseSelectedRectangleChangedEvent( new ROI( oSelectedRectangle ) );
            return true;
        }

        public bool SetBaseRectHeightForSelectedRect ( int iHeight )
        {
            int iIndexOfSelectedRect = oArrayOfRectangles.IndexOf( oSelectedRectangle );
            if ( iIndexOfSelectedRect == -1 )
            {
                return false;
            }
            else
            {
                oArrayOfRectangles[ iIndexOfSelectedRect ].SetBaseRectHeight( iHeight, m_iPercentZoom );
                oSelectedRectangle = oArrayOfRectangles[ iIndexOfSelectedRect ];
            }
            this.pictureBox.Invalidate();
            OnRaiseSelectedRectangleChangedEvent( new ROI( oSelectedRectangle ) );
            return true;
        }

        public bool SetBaseRectWidthForSelectedRect ( int iWidth )
        {
            int iIndexOfSelectedRect = oArrayOfRectangles.IndexOf( oSelectedRectangle );
            if ( iIndexOfSelectedRect == -1 )
            {
                return false;
            }
            else
            {
                oArrayOfRectangles[ iIndexOfSelectedRect ].SetBaseRectWidth( iWidth, m_iPercentZoom );
                oSelectedRectangle = oArrayOfRectangles[ iIndexOfSelectedRect ];
            }
            OnRaiseSelectedRectangleChangedEvent( new ROI( oSelectedRectangle ) );
            this.pictureBox.Invalidate();
            return true;
        }

        public bool SetNameForSelectedRect ( string name )
        {
            return SetValuesForSelectedRect( true, name, false, 0, false, 0, false, 0 );
        }

        public bool SetMinAreaForSelectedRect ( double dMin )
        {
            return SetValuesForSelectedRect( false, "", true, dMin, false, 0, false, 0 );
        }

        public bool SetMinWidthForSelectedRect ( double dMin )
        {
            return SetValuesForSelectedRect( false, "", false, 0, true, dMin, false, 0 );
        }
        public bool SetMinHeightForSelectedRect ( double dMin )
        {
            return SetValuesForSelectedRect( false, "", false, 0, false, 0, true, dMin );
        }

        public bool SetValuesForSelectedRect ( bool bSetName, string name, bool bSetMinArea, double dMinArea, bool bSetMinWidth, double dMinWidth, bool bSetMinHeight, double dMinHeight )
        {

            int iIndexOfSelectedRect = oArrayOfRectangles.IndexOf( oSelectedRectangle );
            if ( iIndexOfSelectedRect == -1 )
            {
                return false;
            }
            else
            {
                if ( bSetName )
                    oArrayOfRectangles[ iIndexOfSelectedRect ].strName = name;
                if ( bSetMinArea )
                    oArrayOfRectangles[ iIndexOfSelectedRect ].dMinArea = dMinArea;
                if ( bSetMinWidth )
                    oArrayOfRectangles[ iIndexOfSelectedRect ].dMinWidth = dMinWidth;
                if ( bSetMinHeight )
                    oArrayOfRectangles[ iIndexOfSelectedRect ].dMinHeight = dMinHeight;

                oSelectedRectangle = oArrayOfRectangles[ iIndexOfSelectedRect ];
            }
            OnRaiseSelectedRectangleChangedEvent( new ROI( oArrayOfRectangles[ iIndexOfSelectedRect ] ) );
            return true;
        }

        public bool SetInspectionTypeForSelectedRect ( ROIInspectionMode enmInspectionMode )
        {
            int iIndexOfSelectedRect = oArrayOfRectangles.IndexOf( oSelectedRectangle );
            if ( iIndexOfSelectedRect == -1 )
            {
                return false;
            }
            else
            {
                oArrayOfRectangles[ iIndexOfSelectedRect ].enmInspectionMode = enmInspectionMode;

                oSelectedRectangle = oArrayOfRectangles[ iIndexOfSelectedRect ];
            }
            OnRaiseSelectedRectangleChangedEvent( new ROI( oArrayOfRectangles[ iIndexOfSelectedRect ] ) );
            this.pictureBox.Invalidate();
            return true;
        }

        public void PanSelectedRectangleVertical ( int iPixelAmount )
        {
            oArrayOfRectangles.Remove( oSelectedRectangle );
            oSelectedRectangle.PanScaledRect( 0, iPixelAmount, m_iPercentZoom );
            oArrayOfRectangles.Add( oSelectedRectangle );
            OnRaiseSelectedRectangleChangedEvent( new ROI( oSelectedRectangle ) );
            this.pictureBox.Invalidate();
        }

        public void PanSelectedRectangleHorizontal ( int iPixelAmount )
        {
            oArrayOfRectangles.Remove( oSelectedRectangle );
            oSelectedRectangle.PanScaledRect( iPixelAmount, 0, m_iPercentZoom );
            oArrayOfRectangles.Add( oSelectedRectangle );
            OnRaiseSelectedRectangleChangedEvent( new ROI( oSelectedRectangle ) );
            this.pictureBox.Invalidate();
        }

        public bool SelectROIByName ( string strROIName )
        {
            foreach ( clsStoredRectangle oStoredRect in this.oArrayOfRectangles )
            {
                if ( oStoredRect.strName == strROIName )
                {
                    oSelectedRectangle = oStoredRect;
                    this.pictureBox.Invalidate();
                    //OnRaiseSelectedRectangleChangedEvent( new ROI( oSelectedRectangle ) );
                    return true;
                }
            }

            return false;
        }
        #endregion
    }
}

 

Please let me know if you have any comments, suggestions, or problems.

Comments

praba
# praba
In this program you have used "using GlueInspectionHMI.Classes"
and "using DMCLabVIEWInterface", So please describe those files also. It will be helpful for me.

Post a comment

Name (required)

Email (required)

CAPTCHA image
Enter the code shown above: