A Place For Answers

Latest

Modern JavaScript Concepts Series – Part 1: Routing

Hey there,

You’ll have to excuse my absence, I’ve been working on a couple of very cool projects. I’d like to share with you guys some of the lessons I took from them. So in the coming months I plan on posting parts of this series, and for the first post we’ll be talking about routing.

Routing refers to the translation of URLs into views, it exists on most modern platforms like .NET MVC and Node.js. In single page applications(from this point on we’ll refer to them as SPA) we can no longer rely on these platforms to handle our views, and we must look elsewhere to find the answer. Many frameworks have rose to answer the issue and they all handle the problem with some aesthetic differences, in principle there are two ways of handling routing. Now before we actually go on to how we do it let’s first describe what we need to make this happen:

  1. A way to start and stop the routing mechanism.
  2. A construct which holds the route and the action to be performed.
  3. Some way to add and remove routes.
  4. And finally the routing translator.

Let’s translate this interface into some code:

router.js:

;(function () {
	function Router () {
		var self = this;
		self.routes = [];

		self.on = function (path, func) {
			// prevent adding same path routes
			var routeIndex = pathFilter(path);
			if (routeIndex == -1) self.routes.push({ route: path, init: func });
		}

		self.off = function (path) {
			// find the correct route and remove it
			var routeIndex = pathFilter(path);
			if (routeIndex != -1) self.routes.splice(routeIndex, 1);
		}

		function routeMatch (path) {
			var routePath = path.match(/\w+(\/\w+)*/i);
			return (routePath == null) ? "" : routePath[0];
		}

		function route (e) {
			// Todo: respond to router event
		}

		self.start = function () {
			// Todo: start listening to router events
		}

		self.stop = function () {
			// Todo: stop listening to router events
		}

		/* Utils */
		function filterByProp (array, prop, val) {
			for (var i = 0; i < array.length; i++) {
				if (array[i][prop] !== undefined && array[i][prop] == val) { return i; }
			};
			return -1;
		}

		function pathFilter (val) {
			// Shorthand for filterByProp()
			return filterByProp(self.routes, "route", val);
		}
	}
	window.Router = Router;
})()

index.html:

<!DOCTYPE html>
<html lang="en">
<head>
	<meta charset="utf-8" name="viewport" content="width=device-width, initial-scale=1.0">
	<title>Router</title>
</head>
<body>
	<a href="#test1">test1</a>
	<a href="#test2">test2</a>
</body>
<script type="text/javascript" src="router.js"></script>
<script type="text/javascript">
	document.addEventListener('DOMContentLoaded',function() {
		var router = new Router()
		router.start()
		router.on( "test1", function () {
			alert("1st hello");
		} )
		router.on( "test2", function () {
			alert("2nd hello");
		} )
	}
</script>
</html>

First Method – Nipping it in the bud

Capturing the page transitions at the source, binding the routing handler to the event that instigates the transition, for this example we’ll be using the anchor tag, but the idea is that whenever we click a routing action it’ll be resolved by our routing translator.

function route (e) {
	var targetPath = routeMatch(this.attributes["href"].value);
	var routeIndex = pathFilter(targetPath);
	if (routeIndex != -1) {
		self.routes[routeIndex].init();
	} else {
		console.log("Error: missing route.");
		// Handle missing route
	}
}

self.start = function () {
	var anchors = document.querySelectorAll("a");
	for (var i = 0; i < anchors.length; i++) {
		anchors[i].addEventListener("click", route, false);
	};
}

self.stop = function () {
	var anchors = document.querySelectorAll("a");
	for (var i = 0; i < anchors.length; i++) {
		anchors[i].removeEventListener("click", route, false);
	};
}

Pros:

  1. Compatibility – This concept should work on most browsers and we all want our code to reach out to as many people, here and there you might need to handle browser quirks if you’re attempting to go legacy but the implementation concept is accessible.

Cons:

  1. DOM Management Required – In this method we’re attaching events to anchor elements, which would have been alright if our pages were static, sadly we’re all about dynamic content today switching views and what nots. That is why we have to call router.stop() and router.start() every time we change the DOM, or alternatively create a new function that handles only the DOM changes so we┬áremove the events as to avoid memory leaks.
  2. History – This method does not take into consideration browser history at all, sometimes that’s a good thing, but I wrote it in the cons cause personally I prefer to have the choice.

 

Second Method – Every action has an opposite and equal reaction

This technique is a bit more modern than our last one, in this method we’re gonna capture the url change or, in case we need to handle browser quirks, the hashbang change( the part in the query string after the # character ).

I feel this method is more inline with how JavaScript works, being an asynchronous language that is fueled by events.

function route (e) {
	var targetPath = routeMatch(location.hash);
	var routeIndex = pathFilter(targetPath);
	if (routeIndex != -1) {
		self.routes[routeIndex].init();
	} else {
		console.log("Error: missing route.");
		// Handle missing route
	}
}

self.start = function () {
	addEventListener("popstate", route, false);
	if( ieUserAgent() || !("onpopstate" in window) ) addEventListener("hashchange", route, false);
}

self.stop = function () {
	removeEventListener("popstate", route, false);
	if( ieUserAgent() || !("onpopstate" in window) ) removeEventListener("hashchange", route, false);
}

</pre>
<pre>/* Utils */
function ieUserAgent () {
 var result = false;
 if ( navigator.userAgent.indexOf("MSIE") > 0 || navigator.userAgent.match(/Trident.*rv\:11\./) != null ) { result = true; }
 return result;
}

Pros:

  1. History – In this method we’re listening directly to the browser’s history and we have it’s state, which is a huge deal for single page application for all sort of reason for instance, we can differentiate between being in the same view(example: being in the 1st view, switching to a 2nd view and then coming back to the 1st view) through it’s state.
  2. Automatic Transition Handling – No DOM management or worry about memory leaks.

Cons:

  1. Compatibility – This method’s greatest drawback, the support for the “popstate” event is growing and all evergreen browsers nowadays support it, we still need to handle IE’s quirks but the support for the event is has around 75%, with opera-mini not supporting at all and safari only partially supporting the event. The “hashchange” event is also not perfect and standing at ~90% with opera-mini still slow on the draw.

 

Now that you’ve seen how both frameworks operate I’ll leave it up to you to decide how you wanna go about, I will say this, most frameworks go for a mix of both methods incorporating all the advantages they can.

 

Next time I plan on talking about data-binding a pretty cool concept, I suggest you stay tuned.

SharePoint 2010: User Profiles Connections

In SharePoint 2010 the social aspect took a step up in the right direction, it allowed users to follow other users much like Twitter and subscribe to what those users do.

You can access that piece of data regarding the user profile using the “Microsoft.Office.Server.UserProfiles” dll, like so:

using (SPSite site = new SPSite("Site URL"))
{
    SPServiceContext ctx = SPServiceContext.GetContext(site);
    UserProfileManager upm = new UserProfileManager(ctx);
    UserProfile profile = upm.GetUserProfile("Domain\\UserName");
    // profile holds all the data about the given user
}

What’s cool about this is the ability to connect people, say Jonathan is following John that’s following Joe.
Now SharePoint 2010 handles the following part pretty nicely and you can find each user, that your user is following like finding out that John is following Joe using:

Colleague[] colleagues = profile.Colleagues.GetItems();

The Colleague class connects a user profile found under Colleague.Profile to the user profile that’s following it, more information can be found here.

But! what if you wanted to go the other way around and find all the user profiles following a specific user?
Well here’s where SharePoint 2010 gets lost, btw SharePoint 2013 does give you a simpler way to grab this data, but here’s where digging around in msdn goes a long way…

You see when a user subscribers to another user profile’s feed, an event occurs, that event can then be scraped from the activity feed using the “Microsoft.Office.Server.ActivityFeed” dll.

ActivityManager am = new ActivityManager(profile);
List<long> bulkRecordIds = new List<long>();
Dictionary<long, MinimalPerson> owners = new Dictionary<long, MinimalPerson>();
Dictionary<long, List<MinimalPerson>> colleaguesOfOwners = new Dictionary<long, List<MinimalPerson>>();

bulkRecordIds.Add(profile.RecordId); // this is only for one user but can easily be translated to multiple users

ActivityFeedGatherer.GetUsersColleaguesAndRights(am, bulkRecordIds, out owners, out colleaguesOfOwners);

At this point, you’ll receive in “colleaguesOfOwners.Keys” a list containing of “MinimalPerson” representation of every user following your man(which you can then translate to a user profile), this way John can find out he’s being followed by Jonathan.

Hope this helps, stay tuned for more posts.

String Base Conversion – for Node.js & JavaScript

This goes out to all you Node.js and JavaScript people out there.
From normal to binary base, we’ll have to turn each string to unicode and then make it binary:

function encode (plainString ) {
	var encodedArray = [];
	for (var i = 0; i &lt; plainString.length; i++) {
		encodedArray.push(plainString.charCodeAt(i).toString(2));
	};
	return encodedArray.join().replace(/,/g,'');
}

Just remember the function does not pad the string with zeros, so for instance “0” will become “110000” and not “00110000”.
Getting back to normal from binary is a bit easier:

function (encodedString) {
	var plainString  = "";
	for (var i = 0; i &lt; encodedString.length; i+=8) {
		var c = String.fromCharCode(parseInt(encodedString.substr(i,8),2));
		plainString = plainString + c;
	};
	return plainString;
}

This bit of code will take every 8 characters,turn it to unicode and then change it to normal characters in normal base.

As always, stay tuned.

JavaScript Matrix Transposition

Lately I found myself using a small piece of code multiple times for multiple projects so I thought I’d share it with you. I wrote this because, although there are code snippets that does this, I needed a version that would support Internet Explorer 7.

The code swaps the rows with the columns and vice versa of a two dimensional array in JavaScript, of course the algorithm can be mimicked in other languages

function transpose(origin) {
    var result = [];
    for (var i = 0; i < origin[0].length; i++) {
        var temp = [];
        for (var j = 0; j < origin.length; j++) {
            temp.push(origin[j][i]);
        };
        result.push(temp);
    };
    return result;
}

If you don’t need support for older browsers though i highly suggest you use this snippet by Shamasis Bhattacharya from his blog:

function transpose(a) {
    return Object.keys(a[0]).map(function (c) {
        return a.map(function (r) {
            return r[c]; 
        });
    });
}

Remember to stay tuned.

Single Page Application – The jQuery Mobile Way

Single page applications are becoming more and more prevalent in these days. So I thought I’d review how it can be achieved using some of today’s libraries and frameworks, starting with my personal first – jQuery Mobile.

Before actually diving into the matter let’s review over the idea that is single page applications. The idea is based on the notion that to get a better user experience you need to remove hindrances, such as waiting times. think about it – what happens when a user travels inside a site? He reads a bit of data in page one, decides to move to page two, reads a little bit there and maybe moves to another page. During all those transitions the browser made about 50-60 requests per page, now some of those requests are files that are shared between all those pages. Single page applications exploit that fact by grouping all the shared files and saving them into browser’s cache or by saving them into the browser’s storage system.

Now that we got over that boring stuff, Let’s dig into the cool stuff. It starts the same as any html page you add the major css and Javascript files:

<!DOCTYPE html>
<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  <link rel="stylesheet" href="css/jquery.mobile-1.3.2.min.css" />
  <script src="js/jquery-1.9.1.min.js"></script>
  <script src="js/jquery.mobile-1.3.2.min.js"></script>
</head>
<body>
</body>
</html>

So here’s where it gets tricky with jQuery Mobile. It all starts with the container:

<div id="page-one" data-role="page" data-title="Page 1">
content goes here
</div>

In that container you can start styling your page and pour content into it, and that’s it whenever you go to that page again you will only see the content of that page. And so, when your done with your main page, you just create another container just like the first with a different id and a title of your choosing. The next step is to create another page and then continue on and on till all your pages are set up. Your HTML page at this point would look something like this:

<!DOCTYPE html>
<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  <link rel="stylesheet" href="css/jquery.mobile-1.3.2.min.css" />
  <link rel="stylesheet" href="css/bootstrap.min.css" media="screen">
  <script src="js/jquery-1.9.1.min.js"></script>
  <script src="js/jquery.mobile-1.3.2.min.js"></script>
</head>
<body>
  <div id="page-one" data-role="page" data-title="Page 1">...</div>
  <div id="page-two" data-role="page" data-title="Page 2">...</div>
</body>
</html>

Thing is, when you view this page you only see the first page so you must be wondering now:
“But Netanel if it’s all one page how do you see and move between the pages?”
Well, that’s simple you just create a hyperlink and redirect to the hashtag with the id of the target page, something like this:

<a href="#page-two">Click here for page #2</a>

The url would be something like this: http://index.html#page-two.
So that’s Single Page Application using the jQuery Mobile, quite simple you have to admit and keeps the mantra of:
img

But wait, what if you want to trigger events on page load, such as querying the server for updated information? well here’s where some jQuery magic comes into place, you can achieve page load events in this fashion with:

$('#page-one').on('pageshow', function(event) {
	alert("page #1");
});

The structure of this SPA is basically this: the shared CSS files, JS files and with them global variables if you want to use them, so you can even pass data between pages, even though that’s never advised, and you know why! Here’s a small diagram to illustrate this:
spadiagram

In the bottom line here, what did we get? We managed to reuse our code, gave the users a better user experience and overall solid structure for our project.

I’ll be updating this post when i finish uploading a nice little example displaying how it’s all tied up with a pretty little bootstrap bow tie on top soon.
Like always keep tuning in for updates.

Big Data Analysis Using Regex & Multi-Threading

What would be appropriate for a first post?

A “Hello World” app right? Luckily, an opportunity presented itself when I was tasked with creating a log analyzer. To give a bit of background on the situation, the system is a server with multiple processors and around 1700 logs, each weighing 10MB.

Like any good little developer, my first mission was to split the problem to smaller bits. The first was how would I recognize the wanted string, well to my aid came – regular expressions or for short regex, to those who don’t know what it is, regex is a standardized way to find and match strings. Almost all programming languages offer regex in some way or form, and there’s a reason why? Cause it’s a great tool.

Getting back to our walk-through, let’s analyze visually a couple of lines from the logs:

20/8/13,13:58:09|17|Admin|DAL.GetEntity|ID='1027234'|Start
20/8/13,13:58:17|26|Admin|DAL.GetEntity|ID='1073947'|Start
20/8/13,13:58:52|26|Admin|DAL.GetEntity|ID='1073947'|End|Success
20/8/13,13:59:03|17|Admin|DAL.GetEntity|ID='1027234'|End|Failure

Now, you can see that there is a pattern going on here, what we want to do is match the function “DAL.GetEntity” at it’s “End” and capture the “ID” and “Success/Failure”. Here’s how I did it with regex:

int successes = 0;
var regex = new Regex(@".*DAL\.GetEntity.*(?<id>\d+).*End\|(?<success>.*)");
String[] lines = File.ReadAllLines(fileName);

foreach (String line in lines)
{
	if (regex.IsMatch(line))
	{
		var matches = regex.Match(line).Groups;
		Boolean success = matches["success"].Value.ToString() == "Success" ? true : false;
		if(success) { successes++; }
	}
}

Now, what do we have here?
The definition of regex, where we describe the pattern, let’s go over some of it:
. – Matches any single character except a newline character.
* – Matches the preceding character zero or more times.
(pattern) – Matches pattern and remembers the match. The matched substring can be retrieved from the resulting Matches collection.
(?<name>pattern) – Matches pattern and gives the match a name.
\d – Matches a digit character. Equivalent to [0-9].
+ – Matches the preceding character one or more times.

More of these can be found here.
While going over the lines, if we find a match, we save the named captured groups of the regex, check it’s values and that’s that – we just analyzed a file… 1 down, ~1699 to go.

Reading all the files synchronously would take a lot of time, by my calculations about a day’s worth of running, that’s where the asynchronously road would shine, having multiple processors sharing the load. The tactic here is basically to perform the analysis above on each file and writing it’s results to another log. To achieve that goal, we’ll create a function that will be in charge of collecting all the file names to be analyzed, open a thread for each of them and letting that thread do all the heavy lifting, here’s how to do it:

String _logDirPath;
String _logName;
int _maxThreads;
Object _lock;
CountdownEvent countdownEvent;
long _successes;
 
public Boolean AnalyzeAllFiles(String logDirectoryPath, String logName = "AnalysisLog", int threadCount = 8)
{
    _successes = 0;
    _logName = logName;
    _logDirPath = logDirectoryPath;
    _maxThreads = threadCount < 1 ? 1 : threadCount;
    _lock = new Object();
    Boolean complete = false;
    try
    {
        List<String> fileNames = Directory.EnumerateFiles(_logDirPath, "log*.txt").ToList();
        countdownEvent = new CountdownEvent(fileNames.Count);
        ThreadPool.SetMaxThreads(_maxThreads, _maxThreads);
        foreach (String filePath in fileNames)
        {
            ThreadPool.QueueUserWorkItem(new WaitCallback(AnalyzeFile), (Object)filePath);
        }
        countdownEvent.Wait();
        lock (_lock)
        {
            using (StreamWriter file = File.AppendText(_logDirPath + "\\" + _logName + ".log"))
            {
                file.WriteLine(String.Format("Successes: {0}", _successes));
            }
        }
        complete = true;
    }
    catch (Exception e)
    {
        Debug.Write(String.Format("{0}", String.IsNullOrEmpty(e.InnerException.Message) ? e.Message : e.InnerException.Message), "LogAnalyzer");
    }
    return complete;
}

Let’s go over a couple of points in the code:

  • In multi-threading we need to lock down resources that are shared amongst multiple threads, so no other thread would try to change that resource, now there’s no need to do this on primitive objects, but a file is definitely not a primitive object that’s why we’re not gonna lock it down but an object that is synced with it.
  • The CountdownEvent class is great cause it let’s us wait and count down till all the threads are complete, very simple and elegant.
  • Using the Threadpool allows us to queue up threads in conjunction with setting the maximum amount of active threads, if we didn’t use it, what would happen is all threads would’ve been created and 1700 threads each polling 10MB files would kill a lot of computers.
  • Queuing up a thread involves instructing that thread which function he needs to perform and we also pass it as a parameter the name of the log file it needs to analyze in the form of an object.

All that’s left is to see how we can combine the regex into the working bee that is the thread:

private void AnalyzeFile(Object p)
{
	String fileName = (String)p;
    StringBuilder results = new StringBuilder();
    try
    {
        String[] lines = File.ReadAllLines(fileName);
    	 
    	foreach (String line in lines)
    	{
    		if (_regex.IsMatch(line))
    		{
    			var matches = _regex.Match(line).Groups;
    			Boolean success = matches["success"].Value == "Success" ? true : false;
    			if (success)
    			{
    				Interlocked.Increment(ref _successes);
    			}
    			results.Append(matches["id"].Value + " " + success + Environment.NewLine);
    		}
    	}
    	lock (_lock)
    	{
    		using (StreamWriter file = File.AppendText(_logDirPath + "\\" + _logName + ".log"))
    		{
    			file.WriteLine(String.Format("File Processed: {0} at: {1}", fileName, DateTime.Now.ToLocalTime().ToString()));
    			file.Write(results.ToString());
    		}
    	}
    }
    catch (Exception e)
    {
        Debug.Write(String.Format("{0}", String.IsNullOrEmpty(e.InnerException.Message) ? e.Message : e.InnerException.Message), "LogAnalyzer");
    }
	countdownEvent.Signal(); // Signal the thread finished
}

Seems simple enough but there’s a few things to note:

  • We need to cast the object parameter to a string to extract the name of the file to be analyzed.
  • Remember I said we didn’t have to lock up primitive objects to change them in a multi-threaded environment? well that wasn’t very exact. We use the Interlocked class to perform the increment as an atomic function, that can’t be interrupted by another thread.
  • Writing to the file is also done with multi-threading in mind, see how we used a StringBuilder to prepare all the data? that was so that the actual writing would take less time, making the lock on the file shorter, allowing other threads to write to the file sooner.
  • In the end we signal to the CountdownEvent: “hey papa process, I’m done here.” what it does behind the scenes it basically decrements the CountdownEvent, when it reaches zero the big process then returns to work.

So before you go on ahead, I want to give you a bit of a comparison:

  • Single-threaded, the process would take about a day’s worth, roughly 24 hours.
  • Multi-threaded with a maximum of 8 active threads, the process finished just a bit over 3 hours.

Here’s the complete code for reference, important to note the code requires .Net Framework 4.0:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading;
 
namespace LogAnalysis
{
    class LogAnalyzer
    {
        String _logDirPath;
        String _logName;
        int _maxThreads;
        Object _lock;
        CountdownEvent countdownEvent;
        Regex _regex = new Regex(@".*DAL\.GetEntity.*(?<id>\d+).*End\|(?<success>.*)");
        long _successes;
 
        public Boolean AnalyzeAllFiles(String logDirectoryPath, String logName = "AnalysisLog", int threadCount = 8)
        {
            _successes = 0;
            _logName = logName;
            _logDirPath = logDirectoryPath;
            _maxThreads = threadCount < 1 ? 1 : threadCount;
            _lock = new Object();
            Boolean complete = false;
            try
            {
                List<String> fileNames = Directory.EnumerateFiles(_logDirPath, "log*.txt").ToList();
                countdownEvent = new CountdownEvent(fileNames.Count);
                ThreadPool.SetMaxThreads(_maxThreads, _maxThreads);
                foreach (String filePath in fileNames)
                {
                    ThreadPool.QueueUserWorkItem(new WaitCallback(AnalyzeFile), (Object)filePath);
                }
                countdownEvent.Wait(); // Wait until all threads completed
                lock (_lock)
                {
                    using (StreamWriter file = File.AppendText(_logDirPath + "\\" + _logName + ".log"))
                    {
                        file.WriteLine(String.Format("Successes: {0}", _successes));
                    }
                }
                complete = true;
            }
            catch (Exception e)
            {
                Debug.Write(String.Format("{0}", String.IsNullOrEmpty(e.InnerException.Message) ? e.Message : e.InnerException.Message), "LogAnalyzer");
            }
            return complete;
        }
        
        private void AnalyzeFile(Object p)
        {
            String fileName = (String)p;
            StringBuilder results = new StringBuilder();
            try
            {
                String[] lines = File.ReadAllLines(fileName);
                 
                foreach (String line in lines)
                {
                    if (_regex.IsMatch(line))
                    {
                        var matches = _regex.Match(line).Groups;
                        Boolean success = matches["success"].Value == "Success" ? true : false;
                        if (success)
                        {
                            Interlocked.Increment(ref _successes);
                        }
                        results.Append(matches["id"].Value + " " + success + Environment.NewLine);
                    }
                }
                lock (_lock)
                {
                    using (StreamWriter file = File.AppendText(_logDirPath + "\\" + _logName + ".log"))
                    {
                        file.WriteLine(String.Format("File Processed: {0} at: {1}", fileName, DateTime.Now.ToLocalTime().ToString()));
                        file.Write(results.ToString());
                    }
                }
            }
            catch (Exception e)
            {
                Debug.Write(String.Format("{0}", String.IsNullOrEmpty(e.InnerException.Message) ? e.Message : e.InnerException.Message), "LogAnalyzer");
            }
            countdownEvent.Signal(); // Signal the thread finished
        }
    }
}

I hope you learned something from this post, stay tuned for more posts.