Thursday, May 14, 2009

How to embed images into XML files



The title may sound confusing.We all know XML as plain text formats while images as binary files.So how can we embed images inside a XML file?Is it some kind of links to the external image files like the web pages?Not exactly.The image is completely embedded inside the nodes of XML in text format as base64 string.

The Base64 Format:


Base64 converts binary data to plain text using 64 case-sensitive, printable ASCII characters: A-Z, a-z, 0-9, plus sign (+) and forward slash (/), and may be terminated with 0-2 "padding" characters represented by the equal sign (=). For example, the eight-byte binary data in hex "35 71 4d 8e 4c 5f db 42″ converts to Base64 text as "NXFNjkxf20I=".

To accomplish this, we need some kind of encoder application that converts the image into the base64 format and embed the text into XML.To extract the image from the XML,we need another application that reads the base64 string,decodes it into the image again.In .NET Framework class library(FCL), we have convenient components in the "System.Convert" namespace to perform this encoding and decoding.Let's look at an example to understand the process.

The Base64 Encoder:

static void Main()
{
string base64FormattedImage= string.empty;
string imageFilePath = @"C:\Images\nature.jpg";
base64FormattedImage= EncodeToBase64FromImage(imageFilePath);

if (!string.IsNullOrEmpty(base64String))
{
string destinationXMLFile = @"C:\XML\NewsContent.xml";
WriteToXML(base64FormattedImage,destinationXMLFile);
}
}

private string EncodeToBase64FromImage(string imageFilePath)
{
System.IO.FileStream inFile;
byte[] binaryData = null;
string base64String=string.Empty;

try
{
inFile = new System.IO.FileStream(imageFilePath,
System.IO.FileMode.Open,
System.IO.FileAccess.Read);
binaryData = new Byte[inFile.Length];
long bytesRead = inFile.Read(binaryData, 0,
(int)inFile.Length);
inFile.Close();
}
catch (System.Exception exp)
{
// Error creating stream or reading from it.
System.Console.WriteLine("{0}", exp.Message);

}
// Convert the binary input into Base64 Encoded output.
try
{
base64String =
System.Convert.ToBase64String(binaryData,
0,
binaryData.Length);
}
catch (System.ArgumentNullException)
{
System.Console.WriteLine("Binary data array is null.");

}

return base64String;
}

private void WriteToXML(string base64FormattedImage,string destinationXMLFile)
{
XmlDocument doc = new XmlDocument();
doc.Load(destinationXMLFile);
XmlNode nd = doc.SelectSingleNode("/contents/content/Images").FirstChild;
nd.InnerText = base64FormattedImage;
doc.Save(destinationXMLFile);
doc = null;
}


The encoder reads an image file from the disk,converts it into Byte array and calls the "System.Convert.ToBase64String" method to get the base64 string from the Byte array.Then main method passes the base64 string to WriteToXML method to save into an XML file.

The XML file looks like this:

<?xml version="1.0" encoding="utf-8"?>
<contents>
<content>
<ID>1</ID>
<Title>The title</Title>
<Body>
Content goes here..
</Body>
<Images>
<Image ID="1">/9j/4AAQSkZJRgABAgEAYABgAAD/4RBY ......</Image>
</Images>
</content>
</contents>


The Base64 Decoder:

static void Main()
{
string sourceXMLFilePath= @"C:\XML\NewsContent.xml";
string destinationImageFilePath = @"C:\Images\nature.jpg";
DecodeFromBase64ToImage(sourceXMLFilePath, destinationImageFilePath);
}

private void DecodeFromBase64ToImage(string sourceXMLFilePath, string destinationImageFilePath)
{
XmlDocument doc = new XmlDocument();
doc.Load(sourceXMLFilePath);
XmlNode nd = doc.SelectSingleNode("/contents/content/Images").FirstChild;
string base64String = nd.InnerText;

byte[] binaryData;
try
{
binaryData =
System.Convert.FromBase64String(base64String);
}
catch (System.ArgumentNullException)
{
System.Console.WriteLine("Base 64 string is null.");
return;
}
catch (System.FormatException)
{
System.Console.WriteLine("Base 64 string length is not " +
"4 or is not an even multiple of 4.");
return;
}

// Write out the decoded data.
System.IO.FileStream outFile;

try
{
outFile = new System.IO.FileStream(destinationImageFilePath,
System.IO.FileMode.Create,
System.IO.FileAccess.Write);
outFile.Write(binaryData, 0, binaryData.Length);
outFile.Close();
}
catch (System.Exception exp)
{
// Error creating stream or writing to it.
System.Console.WriteLine("{0}", exp.Message);
}
}


The decoder parses the XML document,retrieves the base64 string,converts it into Byte Array by calling the "System.Convert.FromBase64String" method.It then saves the Byte Array as an image file into the disk.

This approach is widely used in the online news media industry.The news media companies exchange articles and images among themselves in XML format.The schema of the XML usually follows the popular NITF(News Industry Text Format) schema or DTD.You can find more about NITF Here.

Monday, May 4, 2009

Creating a Private Ruby Gem Server



While installing a new ruby gem in a machine,we normally use the gem install <gem name> command.This command downloads the latest version of a particular gem.The problem is that ROR gems are not always backward compatible.So if a ROR application was developed using a particular version of a gem (Ex:0.1.1) and the latest version of the gem in the internet (normally rubyforge.org) is 0.1.2,problem may arise if the version 0.1.2 is not backward compatible with version 0.1.1.This may cause a serious trouble when the application is deployed into the web server and the wrong version of the gem is installed.Most probably the application will not function as expected.

A solution to this problem is to deploy the exact version of all the gems those were used during development.One way to implement this is hosting the correct version of the gems into a web server(thus call it a private gem server).I have tested this option in a Apache web server.The steps are as follows:

1.Create a directory for hosting gems on the public files area of the web server.Let's refer is as <Base Directory>.

ssh user@web.server
cd /var/www
mkdir my_gem_server


2.Create a sub directory called "gems" under the <Base Directory>.The name of the sub directory must be "gems" by convention.

ssh user@web.server
cd /var/www/my_gem_server
mkdir gems


3.Copy all the necessary gems from the development machine into the /var/www/my_gem_server/gems sub directory.

4.Generate the gem index:

gem comes with a command generate_index which generates all of the files necessary for serving gems over HTTP.Run this command into the <Base Directory> (/var/www/my_gem_server).

gem generate_index -d /var/www/my_gem_server

Now the private gem server is ready to serve the gems for download and install.

To install the private gems into the target web server where the ROR application will be deployed,log in and run the following command.

gem install <gem name> --source http://<private gem server>/<Base Directory>

The correct version of the gems will be installed and thus the application integrity will be maintained.We need to rerun the gem generate_index command each time we add or remove a gem from the private gem server.

Wednesday, April 15, 2009

Replicate the SQL IN keyword in LINQ to SQL



LINQ To SQL has made the life of .NET developers easier by replicating most of the familiar SQL keywords (SELECT,FROM,WHERE,JOIN..) within the scope of the language.It is easier to construct a LINQ to SQL expression for the developers who are used to in raw SQL syntax against different databases.One of the important SQL keyword is "IN" which is used frequently in the "WHERE" clause of SQL.

For example:

SELECT CustomerID,Name FROM Customers WHERE City IN ('Dhaka','Rajshahi','Chittagong','Khulna')

While constructing this query in LINQ to SQL, the usual solution may come to the mind like this:

C#

string listOfCity = "Dhaka,Rajshahi,Chittagong,Khulna";

MyDataContext context = new MyDataContext ();

var result = from cust in context.Customers
where cust.City in listOfCity
select cust.CustomerID,cust.Name;

return result.ToList();


Actually the above expression does not work and with a little
twist we can make it up and running.The correct code is:

List<string> listOfCity = new List<string>();

listOfCity.Add("Dhaka");
listOfCity.Add("Rajshahi");
listOfCity.Add("Chittagong");
listOfCity.Add("Khulna");

/* We can also use array of string as the following

string[] listOfCity = { "Dhaka","Rajshahi","Chittagong","Khulna" }; */

MyDataContext context = new MyDataContext ();

var result = from cust in context.Customers
where listOfCity.Contains(cust.City)
select cust.CustomerID,cust.Name ;

return result.ToList();


The trick is that we define a generic list of string or an array of string.Then we use the <List>.Contains(<Database column name>) syntax to replicate the SQL "IN" syntax in LINQ To SQL.

Monday, April 6, 2009

Decision making in SSIS depending upon condition



SSIS(SQL Server Integration Service) is not much flexible in the case of decision making based on a condition. In many cases you feel the need of a “Condition” block that can be used in the control flow to decide the alternative paths. For example if you want to copy a file named “AnyFile.txt” into a folder named “Current” if the file does not exist in a folder named “Previous”, otherwise move the file to a folder named “Archive”, there is no direct way to test the “If File Exists in Previous Folder” condition. One possible solution is to use the “Script Task”. The steps are as follows:

1. Declare a package level Boolean variable named “IsFileExistsInPrevious” for example.

2. Create a script task and pass the “IsFileExistsInPrevious” variable as “ReadWriteVariable” into it. Pass other “ReadOnly” variables that contain the name of the file if necessary.

3. Using the components in the .NET “System.IO” namespace, check the existence of the file in the “Previous” folder. If the file exists, set the value of the “IsFileExistsInPrevious” variables to “True”. Otherwise set it to “False”.

VB .NET Script:

Dim srcFilePath As String = ""

If File.Exists(srcFilePath ) Then
Dts.Variables("IsFileExistsInPrevious").Value = True
Else
Dts.Variables("IsFileExistsInPrevious").Value = False

End If
Dts.TaskResult = Dts.Results.Success


4. Declare two “File System” tasks that copies files to the “Current” or “Archive” folders depending on the result of the previous “Script Task”. Drag the “Precedence Constraint” line from the previous “Script Task” to any of the following “File System” tasks.

5. Right click the “Precedence Constraint” line and Select “Edit”. In the “Precedence Constraint Editor” window, Select the value of “Evaluation Operation” to “Expression”. Type the following into the “Expression” box.

@[User::IsFileExistsInPrevious]==false

6. Drag the “Precedence Constraint line to the “File System” task that copies the file to the “Current” folder.

7. Right click the “Precedence Constraint” line again and Select “Edit”. In the “Precedence Constraint Editor” window, Select the value of “Evaluation Operation” to “Expression”. Type the following into the “Expression” box.

@[User::IsFileExistsInPrevious]==true

8. Drag this “Precedence Constraint line to the “File System” task that copies the file to the “Archive” folder.

The key point to note here is the usage of the “Precedence Constraint” editor.The “Expression” option enables to select alternative paths depending on condition (the value of a user variable “IsFileExistsInPrevious” in this case).

Tuesday, March 31, 2009

Setting the FTP password from external configuration sources in a SSIS package



SSIS(SQL Server Integration Service) is a great ETL(Extract Transform Load) tool from Microsoft.It has many useful features for integrating data from different sources.One of the handy feature is the "FTP Task" that enables downloading files from a remote FTP address. The "FTP Task" uses a "FTP Connection Manager" component that stores the FTP credentials(URL,User,Password..). Instead of hard coding the credentials into the component, we can use “Expressions” feature of SSIS to set those properties from configuration files,registry entries or environment variables. So we can set the FTP URL,port,user name from the configuration sources.Unfortunately and surprisingly there is no option in the “Expressions” list to set the FTP password.The “Server Password” option just does not appear in the “Expression” List.One work around of this problem is to use the “Script Task” component and set the “Server Password” property of the “FTP Connection Manager” component. The VB .NET code (SQL 2005 only supports VB .NET) snippet below does the following:

1.Accesses the “FTP Connection Manager” component.
2.Sets the “ServerPassword” property with the value of a package variable(“FTPPassword” here).The package variable can be set from any external configuration sources supported by SSIS(XML file,Environment Variable,Registry..).


Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime

Public Class ScriptMain

Public Sub Main()

Dim FTPConnectionManager As ConnectionManager


FTPConnectionManager = Dts.Connections("FTP_Conn")


FTPConnectionManager.Properties("ServerPassword").SetValue(FTPConnectionManager,
Dts.Variables("FTPPassword").Value)

Dts.TaskResult = Dts.Results.Success

End Sub

End Class


Note:The “Script Task” component must execute before the “FTP Task” component .

Monday, March 16, 2009

Adding a serial number field to the ASP .NET GridView



I was developing a page that displays some data from database in a ASP .NET GridView control through LINQ. The grid required to display a serial number field as the first column of each row.Since i was binding the result of a LINQ call(List<SomeObject>) to the grid, i had to add a new field in the "Entity/Model" Class itself and populate the data somehow before binding it to the Grid. I was not willing to add a "Serial #" field into the class only to display a serial number and was looking for a solution at the View Layer.Then i found this handy solution built into the "GridView" control itself.

<asp:gridview id="myGridView" runat="server" autogeneratecolumns="False">

<columns>
<asp:templatefield headertext="#" width="15">
<itemtemplate>
<%# Container.DataItemIndex + 1 %>
</itemtemplate>
</asp:templatefield>
</columns>
</asp:gridview>

The "Container.DataItemIndex" property does the trick and provides an index for each row item in the GridView. Since the index starts from 0, I only had to increase it by 1 to display a serial number to every row without altering the Data Access Layer. Quite useful isn't it ?

Wednesday, February 18, 2009

Passing security credentials to C# 2.0 Web Service using SOAP Header



In a project ,we had a web service that required the client to pass security credentials (user id and password) while calling some of the web methods. First we thought to add two additional parameters (user id and password) to every web method that required authentication. This strategy looked very repetitive and made the method parameter list longer. After exploring for a while we discovered a nice solution of passing credentials through SOAP header. To implement this we need to perform the following steps:

Web Service:

1. Inherit a class from the “System.Web.Services.Protocols.SoapHeader” class that contains the security credential related fields. Put this class into the same file that contains the web service class.

public class SecurityInfo : SoapHeader
{
public string UserID;
public string Password;
}

2. Add a public field into the web service class of the type “SecurityInfo”.

public SecurityInfo SecurityCredentials = new SecurityInfo();

3. Add a “SoapHeader” attribute to the web method that requires access to security credentials passed by the client application. The parameter name of the “SoapHeader” attribute must be same as the public field declared in the above step (“SecurityCredentials” in this case).


[WebMethod(Description = "A method that requires authentication")]
[SoapHeader("SecurityCredentials")]
public void PerformASensitiveTask()
{
//Call the authentication method here with the security credential.

Authenticate(SecurityCredentials);

}

[WebMethod(Description = "Another method that requires
authentication")]
[SoapHeader("SecurityCredentials")]
public void PerformAnotherSensitiveTask()
{
//Call the authentication method here with the security credential.
Authenticate(SecurityCredentials);

}


private void Authenticate(SecurityInfo credentials)
{
// validation logic goes here. An exception is thrown if not validated.
string userId = credentials.UserID;
string password = credentials.Password;
//.....
}


So the final web service code looks like this:


public class TestService : System.Web.Services.WebService
{
public SecurityInfo SecurityCredentials = new SecurityInfo();
public TestService()
{
}

[WebMethod(Description = "A method that requires authentication")]
[SoapHeader("SecurityCredentials")]
public void PerformASensitiveTask()
{
//Call the authentication method here with the security credential.
Authenticate(SecurityCredentials);
}

[WebMethod(Description = "Another method that requires authentication")]
[SoapHeader("SecurityCredentials")]
public void PerformAnotherSensitiveTask()
{
//Call the authentication method here with the security credential.
Authenticate(SecurityCredentials);
}

private void Authenticate(SecurityInfo credentials)
{
// validation logic goes here. An exception is thrown if not validated.
string userId = credentials.UserID;
string password = credentials.Password;
//.....
}

}

public class SecurityInfo : SoapHeader
{
public string UserID;
public string Password;
}


Web Service Consumer (Console App):

After adding a web reference to the web service, Visual Studio automatically adds a “SecurityInfoValue” property to the consumer-side proxy. The security credential class “SecurityInfo” is also exposed to the client application.We just need to instantiate an object of this class,set the credentials and set it into the “SecurityInfoValue” property of the client side proxy class of the web service.We can call multiple web methods after that with the same instance in the current scope.All the web methods will have access to the security credentials.The client code looks like this:



static void Main(string[] args)
{
TestWebService.TestService srvc = new
ServiceTest.TestWebService.TestService();

TestWebService.SecurityInfo sec = new
TestWebService.SecurityInfo();

sec.UserID = "ID";
sec.Password = "Password";
srvc.SecurityInfoValue = sec;

srvc.PerformASensitiveTask();

srvc.PerformAnotherSensitiveTask();

}


I think using “Soap Header” to pass credentials to multiple web methods of a web service is really an elegant solution.

Friday, February 6, 2009

When to apply OOAD concepts and Design Patterns?



I strongly believe that the concepts like OOAD (Object Oriented Analysis and Design) and Design Patterns contribute to make an application well designed and flexible in a very positive way. But over the years I have seen myself and some of my colleagues struggling to decide where to start first in the software design phase when a new project arrives.

Should we list all the memorized design patterns first and search in the requirement list for the scope to apply those?

Should we think of the basic OO concepts like encapsulation, polymorphism or inheritance (“is-a” or “has-a”) and start designing the classes according to those at the first place ?

Often many developers loose the way on their inability to prioritize these processes and thinks too much of theoretical Analysis and Design without concentrating on solving the problem on hand first. I have also faced this problem which is popularly termed as “Analysis Paralysis” in the Software Analysis and Design Books.

I have found an excellent guideline of prioritizing these processes in the book “Head First Object Oriented Analysis and Design” by O’Reilly publication. The guideline suggests that:

1. We should concentrate on solving the problem on hand first with the best possible algorithm and make sure that the software does exactly what the customer expects from it. In a nutshell, if the customer does not recognize the validity of the software, all hard work on Analysis, design, patterns, best practices, latest and greatest technologies gone in vain my friend.

2. After making the software workable, we should apply basic OO principles to make the software flexible by eliminating the duplicate codes and improving a bad design.

3. Finally we should strive for applying proven design patterns so that the software is easy to extend and reuse.


I just wanted to share these principles and priorities with you and believe that remembering the priorities will be helpful to choose the right path while designing and developing software.

Thursday, January 8, 2009

Tracking the completion of a file upload or copy using .NET FileSystemWatcher component



I had an assignment to process the files uploaded by users into a FTP location.Files used be removed removed after the processing was complete.I had developed a .NET Windows Service in the FTP server machine to watch the FTP directory for the uploaded files.The service used to track the files as soon uploaded and start processing.I had used the .NET "FileSystemWatcher" component to monitor the directory.The problem was that,as soon the file was created in the directory(the upload by user had just started), the service used to start processing it.Normally it takes a while for the completion of file upload specially if the size is larger.So the service was processing incomplete files before the upload is complete.After exploring the property,methods and events of the extremely useful "FileSystemWatcher" component, i found the solution to track when the file is completely uploaded.

To mark the upload completion, a temporary file with a ".flag" extension used to be created that matches exactly the name of the uploaded file.For example if the file name is "Data.zip", the name of the temporary file would be "Data.zip.flag".Any valid and unused extension other than ".flag" could have been used here.The service code is now changed to look for the files with ".flag" extension to understand that the file is fully uploaded or copied.It then extracts the file name from the ".flag" file name and starts the normal processing.

Here is the code:

static void Main(string[] args)
{
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = "C:\\Temp";
watcher.Filter = "*.zip";
watcher.NotifyFilter = (NotifyFilters.LastWrite);
watcher.Changed += new FileSystemEventHandler(watcher_Changed);
watcher.EnableRaisingEvents = true;
Console.ReadLine();
}

static void watcher_Changed(object sender, FileSystemEventArgs e)
{
FileInfo fi = new FileInfo(e.FullPath);

if (fi.Length > 0 && !File.Exists(e.FullPath + ".flag"))
{

File.Create(e.FullPath + ".flag");

}

}

The "path" property of "FileSystemWatcher" component is used to mention the directory to watch for and the "filter" property is used to mention file name or file types.But the crucial property for this particular task is the "NotifyFilter" property that is set to "NotifyFilters.LastWrite" here. This will enable the component to notify when the final byte of the file is written to disk.

The "changed" event of the component is handled to get notified when the file upload is complete.Here we have checked the file size to be sure that it is not zero and also checking whether the temporary file exists.If the file has a size(greater than zero) and temporary file does not exist, we are creating the temporary file.The same event is fired multiple times by .NET Framework.So the checking is being done to create the temporary file only once.

Wednesday, January 7, 2009

Retrieve the name of the currently executing method inside the same method.



In one of my projects, the requirement was to log the error information inside a method if an exception occurs. I needed to record the current method name along with the error message. The code was like this:
public void DoSomething()
{
try
{
//The method code goes here.
}
catch (Exception ex)
{
WriteLog("DoSomething", ex.ToString());
//The first parameter of WriteLog method is the name of the current method.
}
}

I was not happy with the way I had to hard code the name of the current method(“DoSomething” here). This way I had to hard code the names of every method where I needed to log the error information.This required to change every method name in the lines of log writing if i had to change the method name.So I was looking for a built in feature that will provide me the current method name.I searched for a while and found a handy solution in the “System.Diagnostics” namespace of the .NET Framework.Here is the code to automatically get the name of the current method.

public void DoSomething()
{
try
{
//The method code goes here.
}
catch (Exception ex)
{
string currentMethodName = new
System.Diagnostics.StackTrace().GetFrame(0).GetMethod().Name;

//The "GetMethod" method returns a "MethodBase" object.With
//this we can access the name of the method and
//other valuable information like access
//modifier,parameters,return types e.tc.//

WriteLog(currentMethodName, ex.ToString());
}
}
I found the API provided by the “System.Diagnostics” namespace really interesting and useful.

Saturday, December 20, 2008

Using the ASP .NET "_doPostBack()" method reference in Javascript



I was assigned to do a task that will need to invoke a javascript method that will perform some client side control state changes and finally submit the page with the "_doPostBack" javascript function. This "_doPostBack" function template is auto generated by the ASP .NET framework and is called when a post back to the server is required as a result of the interaction between the user with the ASP .NET Server Controls(Ex:button click, drop down list selection changes) .

I had first gone for the easy solution and hard coded the function call like this:

<script>
function DoSomeTask()
{
//Do some client side tasks here.
__doPostBack('<control>','');//posts the page back
}
</script>

I then thought should i rely on the hard coded "__doPostBack" method call? What if This function is not generated by ASP .NET under certain conditions? What if the name of the function differs in different browsers? What if Microsoft replaces the function name in the next version of ASP .NET (this application was developed with 1.1) ?

So i searched on the net and found a nice solution. At the server side ,the "GetPostBackClientHyperlink" method of the "Page" object returns the correct script block that does the post back for the ASP .NET server controls. The syntax is:

Page.GetPostBackClientHyperlink(<control>,<argument>);

So i changed the previous javascript block as:

<script>
function DoSomeTask()
{
//Do some client side tasks here.
<%= GetDoPostBackMethodSignature() %>

}
</script>

Where the "GetDoPostBackMethodSignature()" method is declared at the code behind page as:

protected string GetDoPostBackMethodSignature()
{
return Page.GetPostBackClientHyperlink(<Server Control Instance >,string.Empty);

}

Now there is no confustion whether the generated function is named "__doPostBack" or "_sueBillGates".

I trust ASP .NET fully(have no other alternative at the moment though:) ) and sure that my client side codes will not complain about not finding the "_doPostBack" function.

Cheers!