Author Archives: Darin

Fun With SQL (OVER and PARTITION clauses)

0
Filed under Hiring, SQL

I’ve been interviewing over the last few weeks for a new position (If anyone has a need for a very experienced .Net developer with some recent Ruby/Rails/Javascript/Coffeescript/JQuery etc chops, shoot me an email!)

At any rate, one company had an interesting SQL challenge.

In a nutshell, given a table with an ID field and a FLAG bit field, find all the consecutive rows where the FLAG field is 1 (ie a bad record).

The trick was that the output needed to look like:

ID RANGE        Error Count
4000-6000       2001

First, lets insert a CSV file into a table….

BULK INSERT Flags
FROM 'c:\dev\darin\sqltest\sqltest.csv'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n',
FIRSTROW = 2
)
GO

They supplied a 1.6 million row sample table, and expected sub 10 second response time. Sooooo, you also might want to add a clustered index to the ID field.

Let’s see some SQL!

With the sample table in place, I could get down to business.

I’ll admit, I had to google the syntax of the OVER and PARTITION clauses in T-SQL. They’re not something I use every day.

First, I needed all the rows, with a group number grouping all the consecutive rows who’s flag value was 1. I used a Common Table Expression for this. I could have used temp tables and it would have resulted in about the same performance, but… what the hey!

WITH parts AS (
   SELECT *, id - ROW_NUMBER() OVER (ORDER BY id) AS grp
   FROM flags
   WHERE flag = 1
)

Next, I needed the count of each group as part of the result set….

WITH counts AS (
   SELECT *, COUNT(*) OVER (PARTITION BY grp) AS cnt
   FROM parts
)

With those two CTEs in place, all that was left was the output Select….

SELECT cast(min(id) as varchar(20)) + '-' + cast(max(id) as varchar(20)) as IDRange,
   cnt as ErrorCount
FROM counts
WHERE cnt >= 1000
GROUP BY grp, cnt
ORDER BY IDRange

In this case, I only cared about blocks of errors of more than 1000 rows. The concatenated SELECT is just to make a nice output format. realistically, I’d probably only select MIN(ID), MAX(ID), CNT

I haven’t used CTE’s a lot, but I’m finding more and more uses for them.

And the OVER clause is something I really need to get more familiar with.

For the 1.6 million row source table, I was getting sub 2 second response times on this query. No doubt, there’s ways to speed this up, but that’s not too shabby.

PhaseShift and Frets On Fire X Song Packs

0
Filed under Arcade, AutoHotKey, Frets On Fire, Games, Guitar, Stepmania

PhaseShift is an excellent freeware (not open source yet, as far as I can tell, but here’s hoping) version of the RockBand genre of games. It’s even more interesting in that it can interoperate with guitar controllers, game drum kits, vocals, keyboards, MIDI, and even step pads (for Dance Dance Revolution style games, all at the same time!).

Very cool stuff.

What’s even better is that it can read both Frets on Fire songs as well as Stepmania format songs.

Mostly.

There are a ton of great song packs available out on the web, and most that I’ve found work seamlessly with PhaseShift. However, I’ve run into a bunch that do not.

The problem appears to be with the NOTES.MID file (the file that contains all the midi notes that is used by the game to show the onscreen “notes”).

After some digging using a hex editor (my favorite right now is Tiny Hexer), I discovered that in all the cases that failed, the NOTES.MID file appears to contain extranous junk. In many cases, it was a string of keyboard key names, in others it was code snippets. I’m guessing that whatever MIDI editor was used to generate those MID files didn’t properly clear garbage out or compact memory.

At any rate, FoFix appears to ignore the extra junk but PhaseShift does not.

Enter EOF

EOF is a song editor for “fretting” songs for use with rhythm games like Frets On Fire and PhaseShift. The latest version even has specific features to take advantage of elements of PhaseShift that don’t exist in Frets On Fire.

I found that simply using FILE – IMPORT MIDI and importing the NOTES.MID file into EOF, then immediately SAVING the file (and clicking YES to the prompt of “The file hasn’t changed, Save Anyway?”), fixed the problem for every single song I tried it on.

It worked so good in fact, that I wrote up a simple script in AutoHotKey to just run through all the files in a directory and perform those magic steps on them

/*
Convert songs via EOF
*/

F12::
Reload
return

F11::
loop, 1 {
    ;match anywhere in title
    SetTitleMatchMode, 2
    IfWinExist \Songs\
    {
        WinActivate
    }
    else
    {
        msgbox No Window
        return
    }

    ;Copy full path name
    sleep 500
    SendPlay ^+C
    sleep 500
    SendPlay ^+C
    ;msgbox %clipboard%

    ;Over to EOF
    IfWinExist EOF -
    {
        WinActivate
    }
    else
    {
        msgbox No EOF
        return
    }

    sleep 500
    SendInput {F6}{F6}
    sleep 500
    SendInput %clipboard%{enter}
    sleep 1000
    Send ^s
    sleep 500
    Send ^s!FS
    sleep 200
    SendInput Y
    sleep 200
    SendInput Y

    ;Back to DirOpus
    IfWinExist \Songs\
    {
        WinActivate
    }
    else
    {
        return
    }

    sleep 500
    SendPlay {down}

}

Change the loop value from 1 to some count if you want to run it through more than one song at a time.

To explain:

The F12 key I’ve mapped to a reload just to make editing this script easier. It’s not necessary for actually running the script.

The F11 key hooks to the secret sauce.

I used the SEARCH feature in Directory Opus to list only the NOTES.MID files from all songs in all subdirectories of a folder I was targeting. If you don’t have DirOpus, you’ll need to recode the script to accommodate some other logic for getting the list of full pathnames to process.

The script starts by activating the DirOpus window and copying the full path to the selected NOTES.MID file.

Then it switches to EOF, (which needs to already be loaded), and performs the FILE – IMPORT function, pasting in the filename of NOTES.MID.

It then immediately does a FILE – SAVE (and supplies a few Y keypresses to answer the “save anyway” prompt.

And finally, it switches back to DirOpen and moves down to the next file.

It’s a hack to be sure. But it worked a treat for me.

Rock on!

Frets On Fire X From Source Code

0
Filed under Games, Python

poseNOTE: In the process of putting together this post, I discovered both the game PhaseShift, and a fantastic theme for it based on RockBand 3. PhaseShift appears to be everything I was looking for in modifying FoFix and a WHOLE LOT MORE. I pressed on with this post for two main reasons:

  1. It might serve to help anyone else who may be interested in continuing on with more development on FoFix
  2. I had a lot of fun getting everything together and just seeing it work from source.

Still, I highly recommend checking out PhaseShift and the themes that are available for it. There seems to be a lot more work going on with it these days than any of the other rhythm games out there.

If you haven’t played it, Frets on Fire is a RockBand clone from back around 2009. You can read more about the genesis of the game here.

It’s written in Python and open source, so there have been a number of mods made to the original game over the years.

By far the best, in my opinion, is the Frets on Fire X mod, usually called FoFix. Tons of great changes and improvements, plus a fantastic looking theme called ‘RockBand’.

The unfortunate part is the FoFix (and Frets on Fire in general) appear to have fallen by the wayside in terms of people working on them. That’s really too bad, as

it’s a fantastic game which works quite seamlessly.

Still, as good it as it, it could always be better, right?

Features

Really, all I set out to do was add code to the song selection list that would show what instruments were available in each song, without having to actually pick the song and then getting a “This song has no drum part” message. Pretty simple stuff. But this is Python, and open source, and while I’ve worked with open source before, Python was a new beast for me to tangle with.

Getting Organized

If you’re like me, you already have a copy of FoFix installed, it’s just that you run the EXE version of the game, and not the source code version.

If you want to make any changes, you’ll need to be running from the source code itself. That’s what all this is about.

First things first though. This all might not be strictly necessary, but I don’t want to wreck my playable install of the game, so…

  1. Copy the entire FoFix installation to somewhere else, I created a folder called FoFix35 (since 3.121 was the current version).  However, If you have a LOT of songs in your \data\songs folder, don’t copy all of them. There’s no need. Just a few will do. So, at this point, you should have 2 root folders somewhere:
    \FoFix (your original Frets On Fire X installation)
    \FoFix35 (the copy, minus most of the songs)
  2. Create a \FoFix35\src folder if it doesn’t already exist (this is where you’ll want the source files)
  3. Create a \FoFix35\installs folder (this is where I downloaded all the necessary files, see the next section)

The Downloads

Download all these files into the installs folder you created above.

First, the code. You’ll likely want the most recent, active version of FoFix, which, as far as I can tell is 3.121

https://codeload.github.com/fofix/fofix/zip/Release_3.121

Python (Duh, right? You’ll need the 2.6 version)

https://www.python.org/ftp/python/2.6.6/python-2.6.6.msi

PIL (Python Imaging Library, you’ll want 1.1.6, not 1.1.7!)

http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.6.exe

PyAudio

http://people.csail.mit.edu/hubert/pyaudio/packages/pyaudio-0.2.4.py26.exe

PyGame (I used the 1.9.2a version, but if that doesn’t work for you, try installing 1.9.1 over it)

http://www.lfd.uci.edu/~gohlke/pythonlibs/zit7bosa/pygame-1.9.2a0.win32-py2.6.exe

http://pygame.org/ftp/pygame-1.9.1.win32-py2.6.msi

PyOpenGL

http://www.lfd.uci.edu/~gohlke/pythonlibs/zit7bosa/PyOpenGL-3.1.0b2.win32-py2.6.exe

PyOpenGL Accelerator

http://www.lfd.uci.edu/~gohlke/pythonlibs/zit7bosa/PyOpenGL-accelerate-3.1.0b2.win32-py2.6.exe

PyWin32

http://softlayer-dal.dl.sourceforge.net/project/pywin32/pywin32/Build216/pywin32-216.win32-py2.6.exe

The Win32 Dependency Pack

https://fofix.googlecode.com/files/fofix-win32-deppack-20130304.zip

And finally, NumPy (Note: you’ll want this version in particular, older versions didn’t work for me)

http://softlayer-dal.dl.sourceforge.net/project/numpy/NumPy/1.8.1/numpy-1.8.1-win32-superpack-python2.6.exe

 

Installing Everything

First, unzip the Source code zip file into the \src folder you created above. When done, you should have

\FoFix35\src\midi

\FoFix35\src\scripts

\FoFix35\src\win32

plus a bunch of *.py files in the \FoFix35\src folder itself

 

Next, dblclick the Python installation exe and work your way down the list above, till you get to the Win32 dependency pack. Just accept the defaults and you should be able to just install each one after the other.

The dependency pack itself is a zip file that needs to be unpacked in a specific spot.

The zip contains a root folder, ‘deps’. This ‘deps’ folder should be unpacked into:

\FoFix35\src\win32

such that when you’re done, you end up with:

\FoFix35\src\win32\deps

and this deps folder contains ‘bin’, ‘include’ and ‘lib’ folders.

Finally, finish up with the numpy package.

Checking it out

If everything has gone well, you should be able to navigate to

\FoFix35\src\scripts

and run

RunFofFromSources.bat

FoFix should compile and open its normal intro window.

If it doesn’t, be sure to check the log file

\FoFix35\src\fofix.log

This should give you a pretty good indication of what’s gone wrong.

Wrapping Up

If things do go wrong, first check the log I mentioned above.

I ran into several issues while working through this process.

  • I got the wrong versions of some libraries, based on outdated information on the web (Yeah, this post might eventually end up in that same boat!).
  • I started with the wrong version of Python (2.4, you need 2.6).
  • NumPy gave me some issues, in that I didn’t have the right version.
  • The Python Imaging Library I had to back out to an older rev to get functioning

I stuck with the 32bit version of everything. It might be possible to get 64 bit working, but I didn’t go that route.

I’ve played FoFix for almost 4 years now, using 2 Ashley Rock Axe wireless controllers, as well as a Red Octane wireless drum controller. Great fun! And now I can experiment with some Python coding to boot!

Viewing CHM files from Network Drives

0
Filed under Hack of the Day, Troubleshooting

Ok. So this is not a particularly new problem. You get an application, which comes with a help file in the CHM (Compressed Hypertext) format. Works great as long as you don’t try and open it from a networked drive. <sigh>

So, what to do?

Well, Microsoft has a pretty complete suggestion here.

But, it’s registry hacks, it’s not pretty, and, it didn’t work for me on a  Win7 system.

The Help and Manual folks put together a pretty interesting alternative here but this is assuming you’ve got a product you’re installing. Not great for a typical user.

A simpler way

After researching this way more than I should have, I ended up with a simple straightforward solution that works reasonably well.

Just zip the CHM file into a zip file in the same location.

Yes that’s it!

image

Notice in the image that I have zipped the 7Zip CHM file into a zip file. I renamed the file as 7-zip.chm.zip, just to remind me that it contains the CHM file.

To read the file, just double click the ZIP file, which, in modern Explorer windows, will open the ZIP directly, then double click the CHM inside the ZIP itself.

When you do that, Explorer will decompress the CHM file to your temp folder and then open THAT file with the CHM viewer. Because that CHM file is located on your local harddrive, the restrictions in place that prevent viewing networked CHM files, won’t take effect and, presto! CHM viewage without any registry hassles.

 

By the way, the 7Zip command line version is an excellent tool (with a great help file<g>) for doing just this sort of thing, if you happen to like command line tools.

Grab it here.

Merry Christmas

0
Filed under Rants

It’s been a long year.

Merry Christmas to everyone.

Here’s to next year.

Fading two Video Windows in WPF

0
Filed under WPF

imageI was discussing some future features of a commercial jukebox application with the author a few days ago and he lamented about what weak video support the library he had used had for video. In particular, he really wanted to be able to cleanly fade one playing video into another, very much along the lines of cross-fading one audio track into another.

I’d done a little bit of that kind of cross fade for a jukebox plugin I’d written a while back called the NowPlayingScreenSaver, so I thought I might have a go at it.

The result is a simple demo app that illustrates some of the basic WPF animation and media playback techniques quite nicely.

Setting Things Up

Fire up VS2010 and create a new WPF project.

For the mainwindow.xaml, define two mediaelement objects:

<Window x:Class="MainWindow"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Title="MainWindow" Height="350" Width="525">
    <Grid>
        <MediaElement Height="210" HorizontalAlignment="Left" Margin="12,12,0,0" Name="media1" VerticalAlignment="Top" Width="316" LoadedBehavior="Manual" />
        <MediaElement Height="210" HorizontalAlignment="Left" Margin="12,12,0,0" Name="media2" VerticalAlignment="Top" Width="316" Opacity="0" LoadedBehavior="Manual" />
    </Grid>
</Window>

Pretty simple stuff. Note that media2 is completely transparent initially (Opacity=0).

For the codebehind in MainWindow.xaml.vb, first, add a few IMPORTS

Imports System.Windows.Media.Animation
Imports System.Windows.Threading

In the MainWindow class, you need to set up a few regional variables to hold the animation storyboards you’ll use, plus a few other bits

Class MainWindow
    Private _StoryBrd1 As Storyboard
    Private _StoryBrd2 As Storyboard

    Private v1up As Boolean = False

    Private WithEvents _tmr As DispatcherTimer

v1up is simply a flag that indicates which mediaelement is currently the “main", upfront” element.

The _tmr is a timer I’ll explain in a bit.

The Loaded Event

When this xaml windows loads, I want to start the media1 element playing, as well as configure a few things.

Private Sub MainWindow_Loaded(sender As Object, e As System.Windows.RoutedEventArgs) Handles Me.Loaded
    '---- register media controls so we can animate them
    Me.RegisterName(media1.Name, media1)
    Me.RegisterName(media2.Name, media2)

    '---- position second mediaElement beneath the first
    media2.Source = New Uri("Video2.avi", UriKind.Relative)

    '---- kickstart
    '     video playing
    media1.Source = New Uri("Video1.avi", UriKind.Relative)
    v1up = True
    media1.Play()
End Sub

First, you have to use RegisterName on the MediaElements because you’ll be including them in an animation storyboard.

Next, set the Source properties for each MediaElement to point to a couple of video files. It doesn’t matter what you use, but I picked a couple that were around 20 seconds long for convenience sake. Whichever videos you pick, be sure to include them in the project and set them to COPY IF NEWER so that they will end up in your output folder, where the app can find them when it loads.

image

Set v1up = true to indicate media1 is the “current” video playing and start it up.

How Long’s That Video?

The whole point of this exercise is to fade one video into another near the end of the first video.

That means several things:

  1. You have to know how long the first video is.
  2. You have to know how long we want the fade to last.
  3. With that, you can determine at what point to start fading the current video out, but at the same time start playing and fading the other video in.

The duration you can get with the MediaElement.NaturalDuration.TimeSpan property. However, you can’t actually read that property till the video is loaded.

As a result, you can’t setup the animation until the MediaOpened event has fired:

Private Sub media_MediaOpened(sender As Object, e As System.Windows.RoutedEventArgs) Handles media1.MediaOpened, media2.MediaOpened
    '---- once the media has opened, setup storyboard
    SetupStoryboard(DirectCast(sender, MediaElement))
End Sub

Notice that I’m handling the MediaOpened event for BOTH of the MediaElements here.

Also, for the MediaOpened event to fire, the media must be closed when it’s complete, so we do that in the MediaEnded event:

Private Sub media_MediaEnded(sender As Object, e As System.Windows.RoutedEventArgs) Handles media1.MediaEnded, media2.MediaEnded
    With DirectCast(sender, MediaElement)
        .Close()
    End With
End Sub

Starting that Second Video

At some point, near the end of playing the first video, you need to begin playing the second video. There are a number of ways you could do this. For this demo, I chose a simple DispatcherTimer:

''' <summary>
''' When this timer fires, we're at the fade out of the visible video
''' so load the "other" media element and start it
''' </summary>
''' <param name="sender"></param>
''' <param name="e"></param>
''' <remarks></remarks>
Private Sub _tmr_Tick(sender As Object, e As System.EventArgs) Handles _tmr.Tick
    _tmr.Stop()

    With If(v1up, media2, media1)
        v1up = Not v1up
        .Play()
    End With
End Sub

Notice that it just determines which mediaelement is the “current playing” element, and switches to the other mediaelement to begin playing it.

Don’t worry about setting up this timer now. You’ll do that in a bit.

The Fade Effect Animation

You’ll notice that the MediaOpened event contains a call to SetupStoryboard, so you need to define that function now.

A StoryBoard in WPF simply describes a sequence of animation steps that will be followed, given a timeline. The full possibilities and features of storyboards and WPF animation are far beyond what I can go into effectively here, but I’ll concentrate on what’s necessary for this effect.

First, the full procedure:

Private Sub SetupStoryboard(Media As MediaElement)
    '---- setup the storyboard object
    Dim StoryBrd = New Storyboard()

    Dim OpacityAnim = New DoubleAnimationUsingKeyFrames
    OpacityAnim.BeginTime = New TimeSpan(0)
    OpacityAnim.Duration = Media.NaturalDuration.TimeSpan
    Dim kf = New SplineDoubleKeyFrame               'keyframe to fade in
    kf.KeyTime = KeyTime.FromPercent(0)
    kf.Value = 0
    OpacityAnim.KeyFrames.Add(kf)
    kf = New SplineDoubleKeyFrame                   'Keyframe at max opacity
    kf.KeyTime = KeyTime.FromPercent(0.2)
    kf.Value = 1
    OpacityAnim.KeyFrames.Add(kf)
    kf = New SplineDoubleKeyFrame
    kf.KeyTime = KeyTime.FromPercent(0.8)           'keyframe to begin fade out
    kf.Value = 1
    OpacityAnim.KeyFrames.Add(kf)
    kf = New SplineDoubleKeyFrame
    kf.KeyTime = KeyTime.FromPercent(1)             'keyframe to be completely faded out
    kf.Value = 0
    OpacityAnim.KeyFrames.Add(kf)

    Storyboard.SetTargetName(OpacityAnim, Media.Name)
    Storyboard.SetTargetProperty(OpacityAnim, New PropertyPath(System.Windows.Controls.Image.OpacityProperty))

    '---- connect it to the storyboard as well
    StoryBrd.Children.Add(OpacityAnim)
    StoryBrd.Begin(Me)

    If v1up Then
        _StoryBrd1 = StoryBrd
    Else
        _StoryBrd2 = StoryBrd
    End If

    '---- set the trigger timer to 80% of duration from now
    _tmr = New DispatcherTimer()
    _tmr.Interval = New TimeSpan(0, 0, 0, 0, Media.NaturalDuration.TimeSpan.TotalMilliseconds * 0.8)
    _tmr.Start()
End Sub

First, create a new StoryBoard object.

Then, create a DoubleAnimationUsingKeyFrames. This is because:

You’ll be animating Opacity, which is a continuous value (as opposed to something like Visibility, which has discreet values). You want the Opacity value to vary smoothly from one point to some other point in the animation sequence. These points are called KeyFrames.

Since the animation will begin immediately, set the BeginTime to 0.

The overall duration of the animation will be the length of the video itself, so use the NaturalDuration property to retrieve that now.

The next, and probably most confusing, part is to define the KeyFrames. To visualize this, imagine that each KeyFrame object represents one distinct point in time. The animation, then, is responsible for changing whatever property you’ll be animating (in this case Opacity), smoothly from one value to another value, between one KeyFrame and the next.

So, define the first KeyFrame as:

Dim kf = New SplineDoubleKeyFrame               'keyframe to fade in
kf.KeyTime = KeyTime.FromPercent(0)
kf.Value = 0
OpacityAnim.KeyFrames.Add(kf)

and the second as:

kf = New SplineDoubleKeyFrame                   'Keyframe at max opacity
kf.KeyTime = KeyTime.FromPercent(0.2)
kf.Value = 1
OpacityAnim.KeyFrames.Add(kf)

And you can see how this means you’ll be changing the Opacity property from a Value of 0 (transparent), to a Value of 1 (opaque), during a timespan starting at the beginning of the video and lasting for 20% (the .2) of the video’s duration.

The other KeyFrames are defined in the same way.

Now, the animation is defined, but you haven’t told WPF what we’re animating, so do that now:

Storyboard.SetTargetName(OpacityAnim, Media.Name)
Storyboard.SetTargetProperty(OpacityAnim, New PropertyPath(System.Windows.Controls.Image.OpacityProperty))

These lines tell WPF that you’ll be applying this animation to a specific MediaElement, and that the animation itself will be effecting the Opacity Property.

StoryBoards can actually contain any number of animations. In our case, there’s only one though. So you need to add the animation to the StoryBoard and then start it running:

StoryBrd.Children.Add(OpacityAnim)
StoryBrd.Begin(Me)

Next, store the StoryBoard you just created in one or the other regional variables (this is because you need to keep the StoryBoard around long enough to allow it to run it’s course).

And finally, since you have the Video duration available at this point, you need to set up the timer to kick off the other video at the 80% mark in this video.

_tmr = New DispatcherTimer()
_tmr.Interval = New TimeSpan(0, 0, 0, 0, Media.NaturalDuration.TimeSpan.TotalMilliseconds * 0.8)
_tmr.Start()

Wrapup

With all that in place, you should be able to run and see one video smoothly fade in, play, then start to fade out as the second video begins to fade in.

Note that as with anything heavily graphical, you may need a fairly powerful PC and graphics card for the fades to be smooth. However, WPF is based on DirectX which is surprisingly capable even on moderate hardware.

What Next?

First, you may want to also animate the Volume property in a similar manner, to fade it in and out along with the video.

Since this is WPF, you can easily apply transforms to squish and fold the video, as well as all sorts of other forms of transitions (fade is just one example).

What can you come up with?

Remote Workers Are Actually More Engaged

0
Filed under Rants, Telecommuting

Interesting article by Scott Edinger over on LifeHacker.

He lists several reasons for this apparent contradiction, and I’ve seen first hand examples of the good and bad in my last 12+ years of on and off remote work.

One element Scott doesn’t touch on is how different remote work experiences can be. I’ve found that often, companies want to support remote workers, but, since they’ve never really done so before, they either can’t or won’t make the mindset changes that come with doing so successfully.

For me, the most successful remote work scenario is when all or virtually all the people involved are working remotely. I call it the Sailor Paradigm. When you’re all in the same boat, you’ll all tend to gravitate to the same mindset and look for ways to make the situation work for the best. And when everyone’s remote, everyone realizes quickly that true communication takes a little extra effort.

On the other hand, when only one or a few people are working remote, everyone that’s stayed in the office can fall back on much easier communication; the water cooler chat, talking shop over lunch, etc. The remote workers get left out and that can often lead to troubles down the road.

Personally, I’ve always felt the ideal solution is one where all the employees (or at least everyone on a team) works remote, but close enough to an office space or to each other to get some face time one a week or so. That’s not always possible, but it does represent the best of both worlds.

Another Contract Down…

0
Filed under Uncategorized

I’ve read that one of the marks of a good programmer is that they’re able to code themselves out of a job. I’m not sure about that, but, I did finish up this contract earlier than expected, so once again, it’s time to get out there and find a new gig.

If you or your company has any needs for an experienced VB guy (VB.net, VB6, or even farther back, but seriously, anyone still use VB3?), .net guy (I can do C#, despite the blog title<g>), SQL guy (I’m particularly proficient in MS-SQL, stored procs, DB design, normalization, index tuning, etc), or even Javascript/jQuery/Knockout/jQueryMobile, etc, give me shout.

Reply here or use the Contact Link at the top to send me a private message.

SED and AWK for the VB Guy

0
Filed under Utilities

Generally speaking, when I need to do a little file manipulation, I usually fire up .net, whip up a little VB.net command line app to do the trick and off I go.

However, a few nights ago, I needed to do some manipulation on a largish (30+meg) xml file. The manipulation itself was fairly simple:

  1. Find a tag in the file
  2. Insert the contents of other files into the target file, right before the tag

However, it was late, and I was feeling a bit lazy, so I googled it.

What I got was almost all the first page results pointing me to SED or AWK.

What’s that?

SED is short of Stream EDitor. Essentially, it’s an app for running a text file through a set of regular expressions and outputting the results.

AWK is short for Aho, Weinberger and Kernighan, the names of the three programmers who originally came up with it. It’s actually a language for processing text. But, any more, it generally refers to the command line application to applying that language to a input file and generating output from it.

Not big on UNIX

Now, I’ve been around long enough to know what SED and AWK are, but I’ve really never actually used them. However, with all these search results pointing that direction, I had to poke around a little more.

You can grab a version of SED for Windows here:

http://gnuwin32.sourceforge.net/packages/sed.htm

And AWK (or GAWK, the gnu version of AWK, get it<g> ) here:

http://gnuwin32.sourceforge.net/packages/gawk.htm

Those pages have tons of excellent resources, as well as examples, all the docs you’d ever want to read, etc.

And these two apps have been around for so long, that, well, a quick Google search will turn up an example of just about anything you’d need to do with them, so I’m not going to muddy up search results any more than to say that they are really handy tools, especially if you know a little bit about regular expressions.

A Windows Observation

However, I would point out one fairly minor nit that I ran into, at least with the above two ports that I tried.

Both work just fine, but SED I found a tad more troublesome to install. The main problem was that it relies on several external DLLs. You can see these dependencies using DependencyWalker:

image

These files need to be in the same folder as the SED.EXE, and they’re all available at the above link. I guess my feeling is that for such a singular tool, these kinds of dependencies should be compiled in. At one point, many many moons ago, it made at least a little sense to reduce your app diskspace requirements by relying on shared dlls and such. But these days, no one cares if an app like this is 150k vs 500k with all the dependencies compiled in.

AWK (or GAWK), on the other hand, has NO dependencies. None. I copied it to my TOOLS folder, which is on my PATH, and viola! Worked right off. Truly an 0-hassle installation.

They both work very similarly, though SED relies mostly on regular expressions, whereas AWK certainly can be used in conjunction with only regular expressions, but also has the full AWK language behind it to boot.

Speed

One note about speed. There’s nothing to note!

Both of these apps were so fast, even against a 33mb input file, that I didn’t even notice they took any time at all. Running them against this file took about the same time as to actually copy the file.

Granted, my needs were simple, and I’m sure more complex expressions would slow things down. But still. That was refreshing.

And that thing I needed it for?

Removing a singular tag from a large XML file automatically:

awk "!/<\/tag\x3E>/" File1.xml >output.xml

Most of the weird look is from:

  • Having to escape the “/” with a “\”
  • Can’t use a “>” in a batch file command line, because it’s interpreted as a “pipe into an output file” command, which I’ve done at the end of the command with “>output.xml”, so I have to escape it as “\x3E”

I suspect I’ll be using it considerably more in my future!

Gotta Love Texas!

0
Filed under Rants

A neighbor down the street picked up a tank a few months back. He’s always driving it around (it’s completely street legal, tagged, plates, etc).

But you almost have to have a spotter riding up top to help tell the driver whether there’s anything in the way or not. That thing’s got some monster blindspots <g>

NeighborsTank

For those inclined, it’s a Swiss Armored Vehicle (I forget the exact name) from the 60’s I believe, but don’t quote me. I know much more about VB than militaria. It’s almost completely restored (yes, I believe even the gun works assuming you can find (or afford) ammo for it.

Not exactly a daily driver, but it’s awesome to see that thing rolling down a suburban street.