MEDC 2007 - Day 1

Actually the first day was yesterday with MIX, I went over there with high-hopes of watching another great key-note by Scott Guthrie, I made the trip from the Monte Carlo to the Venetian early that morning, got registered and headed up to the meeting room.  I was greeted by a not-so-friendly security guard saying the MIX sticker on my badge wouldn't let me onto the fourth floor...guess there was something in the fine print, oh well.  I came back that afternoon and went to a couple great sessions, one on identity, it really seems like OpenID is really going to merit a series investment in time over the coming months.  The other conference I went to was on security with AJAX enabled applications.  This was a great session as a refresher/reminder that building a secure application really needs to be started from day one and built into the architecture, at least its much easier that way.  Overall the $395 I paid to add MIX to the MEDC conference was well worth it

The conference started out with a bang with the Key Note by Robbie Bach the president of Microsoft’s Entertainment & Devices Division.  He made a great case for developing software as a service and one of the biggest clients for those services is going to be mobile devices.  I also so some great demos on another one of the things I love to work on in my spare time (what's that?) Home Automation.  I really need to look at some of the latest developments in this area starting with Microsoft's Home Server.

Key points on what I took away from today's sessions

• Microsoft/Verisign finally came up with a "sane" way of signing mobile CAB files for WM 5.0.  Before you had to sign all your files, send them to Verisign, let them do their thing, get those files back.  Package them into a CAB and send it back up to them to sign the completed package.  What a pain in the ass, no real way to automate.  Just introduced this week, you can just start sending up the CAB file and it only costs one signing event!  Now it seems reasonable to sign the files...
• There is now an applications out there called fakeGPS.  I have a project coming up in May where I need to report on GPS data from a device, I'm looking forward to digging into this technology.
• According to Microsoft, WM6.0 and the Compact Frameworks are 100% back-wards compatible.  I have a fairly large WM5.0 application I've developed so I'll put this through the test, I really hope this is the case!
• On WM6.0 devices the Compact Frameworks 2.0 SP2 and SQL Server Compact edition are baked into the ROM...nice!
• On the large WM5.0 application I developed there is a good chunk of the code that deals with one handed operation on a Pocket PC type device (read not smart phone) I always thought this was a real hack...it was nice to see the Microsoft presenters give a demo of how to do this (and their implementation was really even more of a hack ;) ). 
• In Windows Mobile there is an event that I can subscribe to (something like Connection Status Changed) this will be fired when the connection via cell phone changes.  As part of my mobile application based upon a schedule I check the server for changes...kind of messy and probably wasteful the easy solution here would be to use an SMS message to kick of replication, yet this isn't that great either.  What I'm thinking about is that I'll build some sort of registration type service where the device can register with the server it's IP address as the connection status changes, then the server could make a TCP/IP call to the device to initiate replication (or maybe just send changes)
• New terminology for WM6.0 (Don't shoot the messenger)
  • Smartphone (no touch pad) - Now called "Windows Mobile Standard"
  • Pocket PC Phone - Now called "Windows Mobile Professional"
  • PDA's (no phone) - Now called "Windows Mobile Classic"

Overall this was a great day...picked up lot's of tips & tricks and validated all my design considerations over the past year.

-ec

64 Bit SSIS & Importing Access Data

Day 1 in Las Vegas for MIX & MEDC 2007...conference starts on Monday, I arrived on Sunday and had planned for a nice relaxing day of enjoying what Las Vegas has to offer.  Only a small bit of work (or at least that's what I thought) I had been manually kicking off a process that imported Shoretel Phone records into a SQL Server 2005 database to do some reporting on phone usage.  I knew this week would be busy so I figured I'd get this setup as a job within SQL Server so I didn't have to worry about it.  I moved my package to the production box created a job with one step that would start the package and send me a status email.  As a good developer I tested it before I headed on my adventure...oh-no it didn't work!

To make a long story short, after screwing around with this for about four hours (which I really couldn't bill my client for) I finally figured out that the OLE DB Jet drivers don't come in a 64 bit flavor...so executing it with the job scheduler didn't turn out to be an option, I finally ended up just creating a simple windows "Task Scheduler" job that kicks off a the 32 bit version of dtexec and life was good...

Here's hoping somebody finds this post and doesn't waste as much time as me!

-ec

The Learning Curve

What is the acceptable learning curve on a new technology?  How much time will you invest in something before you realize you are spinning your wheels and you need to get some actual work done?  I recently had a reporting project as that as I like to say just "Screamed" for an OLAP type solution.  I had considerable experience with MS Reporting Analysis Server 2000 building numerous cubes that just worked great.  My past couple of years (at least since the release of SQL Server 2005) just didn't present any needs for this.  This new project was a perfect excuse to pick up the 2005 Analysis server technology, however this project just wasn't that big and didn't have a lot of hours allocated for it.  I figured, eh this new technology can't be that much different so let's give it a try.  I had bought the books a while ago and took them out over the weekend to the back porch lit up a cigar and read what I thought were the important chapters.  Then I went back in and fired up VS.NET found the right project templates and got started.  This is certainly not a post on how to use MSAS (Microsoft Analysis Server) 2005, but I created my data sources, created my data source views and was off to the races.  Then I created my cubes via the Cube Wizard and I just couldn't get them to compile.  I put the errors I was getting into Google and started the research process, I made incremental gains but it really was a struggle, a lot of new concepts and I didn't have the luxury of any type of formal training.  After a few hours of struggling I was starting to think; well, I'm not going to bill my client for my learning on this technology so do I just need to create some simple stored procs with some aggregate queries and get this project done with the old tried and true big frickin' hammer approach.  I thought just a little big longer, then all the pieces started to fall in to place, than all those pieces started to make a TON of sense.  This solution that MS came up with in their 2005 Analysis Server product release just really kicks ass!  Bottom line is that with the 2000 release of the product it was kind-of like building (or maybe should I say hacking) as solution in VB 6.0 or Access, with MSAS 2005 we have a formal development environment to build very tight well engineered cubes.

So looking back on this at the point I was thinking about abandoning this learning process I was probably 80% of the way there.  I'm very glad I kept with this, overcome the learning curve and have a new tool in my toolbox.

What did I take away from this exercise? First, even if something seems a little difficult and it may be easier to fall back on your existing tool set to solve the problem don't give up on the new technology too soon.  What you are doing is making an investment in your future, making your self more valuable and effective as a developer.  Bringing the right tool to the right problem is critical to solving technical problems today.  Yes, I could have probably built these reports using simple SQL, however with the extra effort and tenacity of not letting this new technology built me resulted in a solution that not only solved the problem at hand but with MSAS solved it quicker (even with the investment in the learning curve).  It also opened a new door of presenting these cubes to the end users so they could use MS Excel to run their own ad-hoc queries against the data and thus present a new set of options to the business to solve future problems. 

Second thing we can learn is when developing our new technologies and frameworks what can we do to reduce (although I don't believe every eliminate) the learning curve associated with other people adopting these technologies.  What can we do to help this?  I'm not sure that building pages upon pages of word documents or web pages is really the right answer here.  I did the perquisite reading and got some of the basic concepts with MSAS but I'm not sure that provided a major benefit.  What we can do is build very simple sample applications, not much more than the typical "Hello World" applications.  Don't get too fancy with these, focus on building small tutorials that are very complete for the topic or concept you want to present but don't go to far to confuse the "learner" with details that don't apply.  Create these tutorials that are somewhat interesting that represent real world problems a bite size chunk at a time.

If you are doing any real productive development chances are you have about 12 hours of work to do but only eight hours of time to get that work done.  What can you do?  Say this the right solutions is just to hard and fall back on your current skill set and spend the 12 hours, work until 10PM, get up the next morning at six and continue the cycle.  Or do you make the investment of your time that builds your tool box to a point where the next time a similar problem comes up, you can bring up the exact right solution and that 12 hour problem gets solved in 4 hours and your solution is exactly the right fit that may also be a building block to solve other business problems.  I'm glad I invested the time in this effort, the day or so I spent getting up to speed on MSAS is already paying back dividends.  Don't give up to soon, and don't always fall back on your tried and true technologies.  Saying all of that, this is where experience comes in, recognizing problems and identifying existing and "real" solutions is just something that good developers have a knack for.  Our customers (either internal or external) don't pay us to play with all the newest and latest technologies they pay us for solutions and before you invest your time you should have a good sense of what you expect from the technology.

Hang in there, there are a ton of new technologies out there, it's going to be a fun ride until us as software engineers have the same basic building blocks as more evolved engineering disciplines such as electrical or mechanical engineering.

Good luck and enjoy your journey!

-ec

Test Driven Development In the Real World

Well its been a couple years now and I've got enough experience doing TDD to feel that I can speak intelligent about what works (at least for me) using this development methodology.  I've seen some people say that they should never write a line of code before writing the tests, well that might be all-and-good for if your primary goal is to get an A on your test this semester, but in the real world, I'm not sure this is the best use of time.  Again, this is speaking from my past experience and the way I develop systems so your mileage may vary.

In my experience, as software developers we have to assume a certain level of stability in the building blocks we create.  Should we create test cases around our business objects that ensure when we execute a SQL Command against the database the SQL Connection object opens the connections and successfully executes the command? 

Let's look at the Red-Green refactor cycle.  At any one time when you are developing software you are attempting to solve a specific problem.  If you aren't sure of the problem you are trying to solve you might not be ready to start coding, it might be time to drop back to the white-board and draw some boxes/puffy clouds, or you may need to revisit your data model.  Either way just getting that little green light to come on within your screen may prove your current API is valid, but I'm guessing that, the API may change a little bit once you continue on in your efforts.

So if you made it this far, you have an open mind and we can continue our discussion...so let me say this, is there a place for TDD?  Abso-fricky-lutely I just did a little effort that was really built as more of a test concurrent type of development effort, but it's almost the same thing.  This was a very complex process where we had a disconnected database on a remote device that synchronized with a server.  We had to create some new "stuff' on the device that all needed to work locally before we could talk to the server.  No problem you say right?  Well where it gets interesting is the fact that the server uses integers to establish the relationships between the tables, that means if I create a new record on the device, I can't use an integer as the primary key, because that will only be available once we look at the identity column on the server database.  We need to build up all our referential integrity on the device database using Guido's then replicate those changes back to the server, create the "real records" send those "real records" back down to the device and make sure all our relationships are still happy.  For about the past year, I've been very happy using NUnit, however the client I'm building this for uses VSTFS so built the testing using this technology...really about the same thing, just slightly different syntax.  My new favorite little utility TestDriven.NET works like a charm with both.  So anyway, I was able to create some really nice scenarios that create the item on the device, add this to it, add that to it, change this, change that, then synchronize with the server and make sure the data I had on the device was the same as after I synchronize.

Another awesome experience I had with TDD was the development of a binary serialization algorithm for DataSets.  This I did as a pure TDD experience, I built my tests for the different data types I needed to serialize, then I created the methods and work on this until each of the unit tests passed.

My other experience wasn't so much with TDD but more of a formalized Unit Testing via NUnit.  I created a VB.NET ASP.NET 2.0 application, with about 150 unit tests, I'm not sure I gained a ton in development efficiency, however it did same my rear-end a couple times with my build cycle.  As part of the continuous integration environment that was in place for this project, we ran the unit tests, if they didn't succeed, we knew we shouldn't deploy.  Worked out slick and once the test were built it worked for "free" so that was kind of nice.

So what's my final opinion on TDD?   Well just as in anything moderation...and finding the right tool to solve the right problem.  I'm not convinced you need to write a test case to make sure the set and get properties work on your business objects, however if you have a complicated workflow, it is certainly worth the investment in time.

-ec

The Deadly Cycle or Keeping up with the Latest Technologies

As we progress through another year as Software Engineers, I'm starting to sense an alarming trend.  Please indulge me as I explain, if you are a software architect you are responsible for picking the right technologies to construct your system, to do this effectively you really need a fairly in-depth knowledge about the existing technologies or building blocks that are not only available today, but what is also available as either a CTP or beta release.  Now you pick the right technology and start constructing your system, let's say it's not a huge system, but it isn't trivial either, let's say you have a cycle time of 6 months.  As you build your system (and you picked the right technology) you will gain the experience necessary to master the technology, and if you planned it right and with a little luck, by the time you are ready to ship, they beta release you were working with go golden so you are set!

Now you are off to your next effort and design and build a project using your newly acquired expertise.  Another 6 month effort, great solution and everyone is happy.  What's happened in the past year?  Well, another new cycle of technology came out requiring even more detailed understanding.  This new technology may be so fundamentally different that your previous experience may now be obsolete.

Let's look at this another way, you know that about 1 year from now a fundamentally new technology is coming out, you need to build something, but you don't want build it off of a technology that will be obsolete when that technology so you hold off for the new technology.  Well your not shipping or even working on your product until you can get a stable CTP, even with that you run the risk of the API's changing by the time your technology goes live.  So you find the right time start constructing your system get a few months into it and -whammo- you see another technology that would be just perfect for your implementation that is really mutually exclusive to the technology you selected.  What do you do?  Just wait another 6 months for that technology to ship?  Probably not...

What are some things we can do to minimize the impact of new technologies?

Keep up with the absolute bleeding edge of technology.  There are many ways to do this.  In the past I only attended one professional conference a year.  Starting last year I attended two conferences www.medc2006.com and www.devconnections.com this year I'll be attending Dev Connections both in the spring and the fall, also MEDC 2007 and MIXX so this year it will be four conferences, with the incredible number of new technologies being release it really only seems like the best way.  With the new technologies coming out there are usually a number of important subjects within that technology.  These conferences aren't really a good way to get a mastery of the technology however it does give a good overview that will let you know the key parts.  These are also good for determining what is real and what isn't.  As to this point, there really isn't a substitute for experience.  How many times have you been to some sort of session or demo where they say "Build an enterprise site within 10 minutes with no lines of code"?  Well I've got news, as Frederick Brooks says in the Mythical Man Month, there's no silver bullet.

Another way that I think we can attempt to isolate ourselves from the rapid change in technology is to focus on the data.  Even if you choose to implement an SOA architecture (what ever you define that to be) when it comes down to it, at some point the rubber is going to meet the road, your going to need some objects with properties and method that actually does the work.  The way you factor these objects may be the biggest factor as to how well you can adopt the latest technology and put the latest and greatest user interface or offer the functionality provided in your objects as services.

Something that may be an option, but I haven't been able to practice is move away from coding.  Even though some people make it look easy building non-trivial software is hard and complex.  Things that look good at 50,000ft start to fall apart if you can't also understand how they work at the 1ft level.  How many times have you worked with the architects in the ivory towers that haven't coded in years and establish the all encompassing architecture that looks good on paper, but when you have to start building those objects with the properties and methods just falls apart or you create way too many lines of code.

On a final note, we need to start thinking about the software develop as business assets, not just lines of codes and assemblies.  What we develop should try to be at the right level of granularity for the task at hand.  If we build our projects where the majority of the business logic is in the UI, well we may have something we can sell and ship (which is by no means a bad thing) but we really don't have something that we can glue together using the latest technologies.

I'm sorry I just can offer any "silver-bullets" here but sometimes recognizing the problem is half the battle.  I think that we as software engineers need do the best we can to keep up with the latest technologies, invest a significant amount of time reading and researching new technologies and last an probably most important keep our customers and end users in mind.  If we do this things will just fall into place, our products will ship and we will start developing the goal of not just developing software assets, but business assets.

-ec

Another Strategy/Tactics Post

As I look over the last 6 years as a consultant, one thing becomes very clear, technical solutions by themselves will only get you so far.  As a great leader once said “Good tactics can save even the worst strategy. Bad tactics will destroy even the best strategy” (George S. Patton) this is very true for software development, but with a slight twist.  As long as you continue an effective tactical operation, the problems with the strategy may remain hidden.

From 2001-2005 I developed a mission critical system for a very large company, over the course of these years, I did what it took (including searching out Internet café’s in Key West) to keep this system operating as efficiently as possible.  As an outside technical development asset, my primary mission was make sure I kept the business constituents happy, the scope of the change I could affect on the organization was limited, for long term, this change would be required to embrace and support system.  As I left this engagement in early 2006, it was clear that this system was not going to be properly supported.  In March of 2007 for better or worse, Software Logistics was re-engaged on this effort and will make a significant positive impact.

So how does this come back to my original quote from GSP?  In this effort I worked on the “Product” and very little on the “Process”.  The product succeeded at the tactical level with assistance of Internet cafés in Key West, however at the strategic level the proper systems and protocols where not being put in place.  It was clear the technical/tactical part of the solution was sound however the strategic part needed some help.   What was required for long term success on this project was a complementary skill set.  Enter my business partner David E. McCracken, I’m extremely excited to work with Dave on this project going forward.   There is a considerable amount of power and flexibility within the system that was developed however this power and flexibility came at a cost of an investment in understanding two domains, the first being a non-trivial understanding of the subject matter data and the second being an understanding of the architecture.  The adoption beyond the immediate team required considerable effort that in some way is mutually exclusive to the day-to-day activities.    This effort is not in creating the code, stored procedures or even the documentation around these software artifacts.  This effort is in making sure that our tactical efforts that we perform as software engineers today not only satisfy the immediate and pressing needs of the business, but also are done is such a way that they can be repeated on a consistent and predictable fashion in to support strategic goals.  In the short term tactics can make up for poor strategics, however in the long run this is not a sustainable model.  Some great gems of knowledge can be found on David's blog on some ideas on how these efforts can take place.

Bottom line is how much are you working to satisfy your business constituent's tactical and immediate needs and how many fail-safes do you have in place to ensure the long term strategic organizational goals?  Do you have the right infrastructure in place for both these equally important journeys?

-ec

DevConnection Orlando 2007 - Day 1

I was up in the air as to going to DevConnections earlier this March, but after making the decision and attending Day 1, I'm extremely glad I showed up.

NiVek J.D.'s Maiden Voyage

Everything came together today for NiVek's J.D.'s official first journey.  NiVek J.D. is a remote control tractor purchased from Target that was converted over to be controlled via a small on-board computer programmed in Java.  The same basic board I used for NiVek I was used for this "robot".

This robot has two cool features not on NiVek I.  The first is it has a GPS module purchased from Parallax.  The second and which I think is really exciting is it using a Windows Mobile device as a "Repeater" that allows for communications from the NiVek embedded computer to a a PC.  This design is based upon software components that are part of WiMo or Windows Mobile Robot, you really need to check out that site!  This consists of some kewl software components that are made up of a Compact Framework 2.0 application that runs on Windows Mobile and a collection of services that run under Microsoft's Robotics Studio.

NiVek J.D.'s Hardware

Basically the radio was just ripped out of the existing remote control tractor and and the motors were connected to the NiVek embedded computer.  As with the "original" WiMo, the Windows Mobile device was mounted on a servo with a CD-ROM.  The original WiMo used a SmartPhone not a Pocket PC so that probably worked out a little better, it's nice to be able to pan the camera however with my driving skills (smacking into walls) one of these days, I'm pretty sure the CD-ROM is going to end up in pieces .  I need to re-think that part of the design.

NiVeK J.D.'s Maiden Voyage (well at least one of the first few)

50,000ft System Overview

• The actual robot itself is controlled by an embedded computer based upon a small PIC processor with some additional components that allow it to be programmed in Java with 32K RAM & ROM this was purchased from Parallax and is called a Javelin Stamp.  (See this post for more information)
• The embedded computer has a BlueTooth transceiver module that allows it to communicate with a Windows Mobile Device.
• The Windows Mobile Device has a Compact Frameworks 2.0 application running code available from the WiMo Bot web site.
• The Windows Mobile WiMo application communicates with the robot via BlueTooth.  It also has the ability to listen on a socket for connections from a remote application.  Since this is a Windows Mobile device (in my case a phone), it will not only work while it's connected via a local LAN via WiFi, but it can also communicate via the GPRS radio and be a sort of "repeater" that will allow it to communicate to a host controller program anywhere it has cellular reception, just think about this...this is very kewl!  A nice feature is on the opening screen shot of WiMo it tells you the IP address of the device.
  
• On the PC side you have a set of Microsoft Robotics Studio services.  When these services first start you will be greeted by a dialog that will allow you to enter the IP address of the remote Windows Mobile device.
  
• Once you press "Connect" (and the software gods are shining on you) you should establish a connection from your PC to the WiMo application.
• At this point a couple of additional forms will show up from the MSRS services.  The one in the upper left is displaying console messages from MSRS (Microsoft Robotics Studio).  This is a great way to see what's actually going on with your services. 
• The one in the upper right is from a service that came with WiMo (with the addition of buttons to control the motors).  Another really cool built in feature with WiMo is the ability to use the camera on your Windows Mobile device to send pictures back from your robot.  This from also sends messages to the core WiMo communications MSRS service to pass those to the WiMo device application.  These messages allow for control of the robot from the PC.
• The dialog in the bottom is an additional MSRS service that was built that subscribes the the TextMessageReceived event from the core WiMo service. 
  • The NiVek embedded computer spits out GPS readings every second (probably need to change this so it only sends when the location changes). 
  • This gets sent from NiVek to the the WiMo software on the device with a simple <stx><etx> binary protocol.  The WiMo software turns it into a simple text message. 
  • For our GPS WiMo constructs a simple text message "GPS: #Sats=4 Lat=28.4.042 Long=82.42.5522". 
  • This text message is sent over the wire from the Windows Mobile application to the WimoComm MSRS service. 
  • The WiMo MSRS service picks up the text message and finds any services that subscribe to this type of incoming event.
  • The GPS Point plotter MSRS service subscribes to these messages so it takes those readings and plots the on the crude dialog you see below.  The challenge here is that for a robot this size of NiVek J.D. if it moves 50ft that's a long distance, and the resolution on the GPS module I purchased just doesn't seem to be all that accurate.

Finally one last picture of you host at the controls!

Looks like my day job is going to be busy over the next few weeks with DevConnections and MEDC but I hope to sneak in a few hours every once in a while to push this effort forward!

Can anyone figure out what NiVek stands for?

- ec