Sphinx With MAMP

August 4, 2010 at 8:29 am · Filed under PHP, tips ·Tagged , , ,

It’s late and I’m pretty damn sleepy so I’ll just do this in quick lists. Today I was trying to install Sphinx on my local machine, a Macbook Pro. I’m using MAMP 1.9 to dev a new project which will involve Sphinx. However, installing Sphinx turned out to be kind of a pain in the ass. Just as a note, my laptop wasn’t running mysql to begin with.

The problem occurred when trying to install Sphinx (latest stable version – 0.9.9) with mysql support. Apparently MAMP 1.9 doesn’t come with all the mysql files that Sphinx needs to install with mysql support. Doing ./configure on Sphinx just kept leading to dead ends and a message asking me to install the mysql-devel package or explicitly specify MySQL library folders and such and specifying various folders inside of MAMP didn’t turn out well.

Now I’m sure there’s a more ‘ninja’ way to do this but this is what worked for me:

  • Install MySQL. Download the DMG from http://dev.mysql.com/downloads/mysql/. Make sure to download the correct one (32-bit vs. 64-bit). After that it’s a surprisingly simple install. You don’t need to start the service or install the MySQLStartupItem.pkg.
  • After that, ./configure on sphinx worked just fine since MySQL and all the various files are in all the standard locations. Make sure to include the path to install to via the –prefix option. The typical Sphinx installation usually goes in /usr/local/sphinx/ so: ./configure –prefix=/usr/local/sphinx
  • Now do make and sudo make install
  • Everything else is fairly straightforward.

Some extra notes:

  • When configuring your sphinx.conf file, make sure to set sql_sock = /Applications/MAMP/tmp/mysql/mysql.sock when editing your sources. Also don’t forget to specify the right port (MAMP defaults to 8889 for MySQL).
  • Not sure why but it kept trying to look for sphinx.conf in /usr/local/etc/sphinx.conf so when running indexer or search I would have to explicitly specify the location of the sphinx.conf file.
  • If you get permission errors when doing indexer it’s probably because you forgot to sudo.

Drop me a comment if this helped you any or if you have other questions!

Leave a Comment

Importing PhotoPost Pro photos to vBulletin Albums

July 4, 2010 at 2:42 am · Filed under PHP

Background
First, a little background on this post. I own a socal mountain biking forum, www.socaltrailriders.org and recently had to upgrade from 3.8.2 to 4.04. Along the way I also updated vbSEO, and our classifieds system to make them compatible with 4.04 which was a major upgrade. However, we were using Photopost Pro for our photo gallery software but I wanted to move to vBulletin’s albums after having some issues with Photopost Pro.

I researched ways to import from Photopost Pro to vBulletin albums but found no solution. Being a dveloper myself, I decided to make one. This would be my first time making any import script for third party software but I figured it couldn’t be too crazy, right?

The Research
I started off by doing some exploration and figuring out how files and database data are laid out for both Photopost and vBulletin’s album system. I had to dig in a little bit to really figure out how both pieces of software really ticked but now have a fairly good grasp of how it’s structured which is critical if you want to make an import script. vBulletin actually had a bit more nuances than Photopost which was fairly straightforward to figure out. For example, for a user with ID of 191, vbulletin stores attachments in the path “[attachment dir]/1/9/1/”.

Validation and Checking

I’m a strong believer in building code that handles all types of situations in a graceful manner. What if Photopost’s database references a file that no longer exists? What if you’re trying to move files somewhere that you don’t have write access to? These are just a couple of situations that you have to account for in an import script. So, I built a ton of it into this import script and tested it until I was very confident that it would handle every situation in an appropriate manner.

“Features”

I say “features” because this is more of an overview of the functionality of my photopost pro -> vb import script. But I’ll just list the ones that come to mind:

  • categories are preserved and can be configured a bit. categories created by members in Photopost will become albums in vBulletin. on our Photopost gallery, we had a series of categories that were shared amongst a lot of members. since vBulletin albums aren’t shared, I coded the import script to create a new album for the user if they had uploaded to one of those common albums.
  • Also, it can import multiple Photopost categories into one album. For example – on our Photopost gallery we had close to ten categories related to bikes which everybody shared. I configured it so that if I uploaded 4 photos across four of those bike categories, it imported those photos into one vbulletin album called “My Bikes” for me.
  • Photo titles and view counts were preserved.
  • Thumbnails were created based on the max attachment width specified inside of vBulletin’s options.
  • Lots of other little things…

What it doesn’t do: preserve comments or permissions.

Do You Need a PhotoPost Pro -> vBulletin Album Import?

I realize some of you out there may want or need to move from PhotoPost Pro to vBulletin so I’ll offer this import as a paid service. Comment or email me for information about that.

albums in vbulletin freshly imported from photopost pro

Comments (1)

The Legend of the Street Programmer

January 25, 2010 at 9:19 pm · Filed under Uncategorized

     A long time ago there was a young upstart programmer who believed firmly in coding standards and the DRY principle. He was in a dev team which, for the most part was fairly productive, despite their manager. One day the dev team met with their seasoned manager who was deeply imbued with the lead-last form of leadership and well versed in the art of stifling progress, as a good seasoned manager should be.

     One day the upstart programmer disagreed with the seasoned manager on the topic of database normalization. The upstart programmer, seeing that the database was unnecessarily repeating too much of the same data, in various tables, wanted to apply DRY principles by adopting at least the first normal form, otherwise known as 1NF.

     Upon hearing the upstart’s argument, the seasoned manager responded with,

“Young upstart, you may read a lot of books, but out in the real world, out on the street, we do things differently.”

     Thus began the legend of the Street Programmer.

     You see, the seasoned manager had real world experience, and in his world, real-world programmers (the type that programmed from the street) simply didn’t have time to read books, or catch up on latest technologies. No, in fact, if they didn’t agree with the purpose of a new concept, they burrowed themselves deeper into their comfortable leather chairs and dismissed all comers for lack of experience.

     The seasoned manager, being a Street Programmer himself, went on to talk down the use of hashed and encoded passwords, the uselessness of Object Oriented Programming, and spoke lovingly of the use of REGISTER GLOBALS in PHP. This was not a man to be trifled with, he was a man of the street, and he knew his business.

     The upstart programmer, having seen the error of his ways, turned his face downward and exited the conference room where the Street Programmer reigned supreme, and subsequently got a new job that though lacking in a sufficient amount of Street Programmers, was still able to make do. The Street Programmer’s company had other ideas, he was able to sell it to marketing types who bought into the depth of his Street Programming knowledge, and he walked away a happy man.

Leave a Comment

More Speed: CSS Sprites!

January 21, 2010 at 6:57 pm · Filed under performance, tips ·Tagged , , ,

Hey folks! This ones for those speed freaks who are never satisfied with their page load times. This is for good reason. Page load time could mean the difference between a conversion and a lost visitor.

Why?

There are many, many things you can do to improve site performance. Today I’ll explain CSS sprites and why they can help with performance. The goal of using CSS sprites is to minimize the number of HTTP requests. Each HTTP requests has overhead – the browser has to send a request to your web server and then the web server has to respond with an answer (which could include an image, css file, etc). So minimize the amount of HTTP requests and you speed up your page.

What’s a CSS Sprite?

What the heck is this CSS sprite thing that I’m talking about? Well you know all those little background images that you put on your webpage via CSS? They may be round corners, gradients, icons, etc. Well each of those comes with the price of HTTP requests. Using CSS sprites is essentially a technique of combining some of those background images into one image (called the sprite) to minimize the number of HTTP requests.

How?

Luckily there is a tool that makes generating this CSS sprite super easy: SpriteMe. It was created by Steve Sounders, a performance guru. Once you create your CSS Sprite, use it in your CSS by utilizing background-position. Using background-position, you can adjust which part of the CSS Sprite shows up, allowing you to reuse the CSS Sprite throughout your page. On a recent project I was involved with, we reduced the number of HTTP requests by 13!
example of css sprite

Interested in more performance tips? Feel free to check out the Yahoo Developer Network, they have some great advice on this subject.

Leave a Comment

CakePHP Integration With Other Apps

January 14, 2010 at 8:17 pm · Filed under PHP, tips

I just had one hell of a time integrating BBPress into a CakePHP site I’m putting together.

See, the problem with integrating something into CakePHP is that it will automatically try to wrap it within the framework, for example, if I want to add a blog, the url I’m shooting for will be: www.mysite.com/blog/, but if I just stick a directory under CakePHP and call it ‘blog’, then it throws up controller errors. The key then is to stick the directory under ‘webroot’, but alas we run into another problem. If you type in the url www.mysite.com/blog/ it will automatically send you to www.mysite.com/app/webroot/blog/, and that’s not pretty.

Where’s the answer then? It’s in your .htaccess file of course! So, I started searching…

At first I started off with the keywords: ‘cakephp’ and ‘subdirectories’, but all I got were tips on how to install CakePHP into a subdirectory, not quite what I was searching for.

I then thought of the types of software other people might look to integrate into their searches, because I was 100% sure I wasn’t the first. I settled on ‘WordPress‘, ‘cakephp’, and ‘integration’. Which landed me on my answer here: WordPress into CakePHP: The right way! I then proceeded to kick myself for not having looked at PlanetCakePHP.org in the first place.

Leave a Comment

Minify! And New Site! Ohnoes!

August 4, 2009 at 3:11 am · Filed under performance, PHP

This one will be quick.

Just got done configuring minify onto my local dev environment and then onto the live environment. It rocks! It was a little fussy about the setup, but just pay close attention to the config.php file and you’ll be fine. It works as advertised and definitely helps on load time.

Oh – and I did this on my new website which I “soft-launched” yesterday. It’s a site that allows you to search for and buy shoes via a super easy to use website. The content is intelligently cached from Amazon’s web services. Keep an eye out for a post going into further detail on the implementation.

Update: CakePHP published my article about Got2BShoes in their Bakery.

Leave a Comment

Easy Pop-Under

June 18, 2009 at 6:16 pm · Filed under Uncategorized ·Tagged , , , ,

Pop-up advertising is nothing new to the web; we’ve been fluently closing free iPod offers for years now.  However, the latest breed of these attempts to be less intrusive by popping under the current browser instead, with the intention of you not seeing the ad until you’ve closed the browser window.   Fellow Mac users probably see the fault in this logic (since after a typical browsing session, you’d hit Apple-Q, which kills all browser windows at the same time), but nevertheless the technique is growing in popularity, and you might be wondering how to do it.

Fortunately, this is a VERY simple technique which can be cross-browser friendly with only two Javascript commands:

// specify the destination and window size
function popUnder(url, height, width) {

     // spawn the new window using blur() to attempt to
    // force the window into the background
    window.open(url,
        'width='+width
        +',height='+height
        +',left=200,top=200'
    ).blur();

     // force this window into the foreground (to cover
    // any browsers that were unresponsive to blur()
    window.focus();
}

And that’s it — you can now be on the road to being a more subtle nuisance. =)

Comments (1)

Google Analytics Anomaly: More Visits than Page Views?

June 4, 2009 at 7:16 pm · Filed under analytics

wtfBy definition, page views on any given site should be always be higher than the number of visits. However, myself and some other people have came across an anomaly in Google Analytics where visits are the one that is higher.

I don’t have a solution to offer yet, however I do have an explanation…

This is related to sub profiles and events. If you have a parent profile with events and make sub profiles from it, you will most likely run into this issue (until it is solved by Google). I’ll first explain events and sub profiles. If you’re already familiar, feel free to skip the next two sections.

Events

Events are not associated with page views at all. That was actually a perk of using an event to keep track of non page view type information. In the past, we had to use trackPageview() to reports that didn’t occur ‘naturally’. The drawback of that is that it created fake page views which inflated our page view numbers. Using events, that information is kept separate from page views, allowing us to gather useful data and still trust our page view figures.

Sub Profiles

In Google Analytics, you have a unique ID for every parent profile you have a unique Web Property ID and a parent profile for that web property. You can then create other profiles for that web property and filter out various data. The filters usually take out traffic to certain URL’s, certain hostnames, etc etc. This is useful when you want to view only a subset of data or want to see an abstraction of the data.

Ohnoes!

the event visitsThe anomaly occurs because filters in sub profiles won’t also filter events. So what? The events are associated with visits. Since events aren’t filtered, the visits come with them. If your parent profile has 562k visits with events (random number) then your sub profile will automatically have 562k visits even if you filtered out ALL traffic data. That’s exactly what happened in the screenshot from the first paragraph.

 

 

Hopefully that saves some head scratching out there. I’ll post again if I find a solution or if I notice that Google takes action on it.

Comments (7)

Skipping an Abstract Generation

May 8, 2009 at 5:59 am · Filed under PHP, tips, tutorials

One of the issues of working with an existing OOP codebase is that the original author may have programmed abstract classes with a fixed class depth in mind. For example:

abstract class animal
{
    abstract function speak();
}

class dog extends animal
{
    function speak() {
        print "woof";
    }
}

class cat extends animal
{
    function speak() {
        print "meow";
    }
}

All is well in this quadroped example, but what if you decide you’d like to break cats down into seperate sub-species, but leave dogs as is? Presuming that you already have hundreds of animal extension classes written, it would be tedious to reconfigure the abstract base class to handle this exception. Luckily, abstract requirements can skip a generation:

abstract class animal
{
    abstract function speak();
}

class dog extends animal
{
    function speak() {
        print "woof";
    }
}

abstract class cat extends animal
{
    // some general cat methods
}

class kitten extends cat
{
    function speak() {
        print "meow";
    }
}

class lion extends cat
{
    function speak() {
        print "roar";
    }
}

The intermediate class needs to be declared as abstract to indicate that any requirements of both itself and its parent(s) are to be passed to its child. Because the child classes, such as kitten and lion, contain the speak() method, they have satisfied the abstract requirements of the grandparent class, animal. Meanwhile, the dog class still functions as before.

Leave a Comment

Sphinx Search: An Appetizer

April 18, 2009 at 8:38 pm · Filed under PHP

Recently I’ve been playing with Sphinx which promises to be a great site search engine solution at my place of work. This post isn’t meant to be a comprehensive tutorial but a brief overview meant to wet your appetite.

What Is Sphinx?

Sphinx is a standalone search engine that can be used to power search capability in many applications. It’s extremely quick, relevant, scalable, and highly configurable.

If you’re trying to create search functionality and using MySQL to do ‘LIKE’ searches, I highly recommend you at least look into using sphinx.

Get Sphinx

Download the source here at this link. My install experience on a Linux machine went very smoothly. It’s simply a matter of unpacking and then doing
./configure, make, make install. I decided to do ./configure --prefix=/usr/local/sphinx as that is the prefix used in the Sphinx documentation.

Here’s a quick rundown on the contents of your installation:

  • [sphinx dir]/etc/ – sphinx configuration file goes here
  • [sphinx dir]/var/data/ – index files
  • [sphinx dir]/bin/ – useful command line tools and the search daemon

Simple enough so far, right?

Setting Up Sphinx

Before you can start searching, you need to edit and create a sphinx.conf file. Go into the [sphinx dir]/etc/ folder and copy the example sphinx configuration file. Go through it and edit it to your hearts desire. Make sure to become good friends with the documentation as it’ll walk you through each and every available option.

The heart of config file is as follows:

  • Define sources. Each source includes an SQL query. This query is the information you want to be searchable. You can even include fields in this query which you declare as attributes. Afterwards, you’ll be able to sort and/or filter by these attributes.
  • Define indices. Each index points to a source and includes various additional options for how the information is searched. There can be multiple indices pointing to the same source. When searching, you have the ability to search one or more specific indices.

But again, make sure to check out Sphinx’s own documentation. After setting up your config file, run the [sphinx bin]/bin/indexer tool to collect the data and make your indices.

Use Sphinx!

To search directly on the index without going through the daemon, use [sphinx dir]/bin/search. Doing this after running indexer is a good idea as it bypasses api’s and daemons, which can be a source of bugs or confusion. You might also want to play with [sphinx dir]/bin/indextool which will give you some information about the indices you just created and along with [sphinx dir]/bin/search, can prove to be great debugging tools.  

If the search looks to be working well, go ahead and turn on the [sphinx dir]/bin/searchd daemon and try the API’s. Currently Sphinx provides API’s for PHP, Python, and Java. My experience was using the PHP version. These API’s have very useful options related to sorting, field weighting, filtering, etc.

Ohnoes – API/daemon trouble!

This caused me hours of head scratching so I’m hoping I can save some people a bit of frustration here. Remember to restart the searchd daemon after you change configuration options! I had issues when I was getting garbage search results from the PHP API after changing sphinx.conf.

Now go and give your users an awesome search experience!

Comments (1)

Older Posts »
Follow

Get every new post delivered to your Inbox.