BadImageFormatException debugging web site running in x64 mode

There are a lot of questions on this forum about the BadImageFormatException, but none quite matches my issue.

I have a solution containing several projects. When I run in Debug or Release + Any CPU, everything works fine. When I switch to x64, however, and run my ASP.NET MVC project, I get the infamous BadImageFormatException

Could not load file or assembly 'AgileEFLib' or one of its dependencies. An attempt was made to load a program with an incorrect format.

AgileEFLib is an in-house utility library, so I have plenty flexibility with that. But the really weird thing is that when I run a console app that also has a dependency on AgileEFLib, it works perfectly. The only time it fails is when I run the web site.

I have checked that all projects in the Release|x64 configuration are set to compile as x64.

What else can I do to troubleshoot this?

EDIT: Another useful bit of information is that this problem only occurs when I'm debugging using Visual Studio's IIS Express. When I publish and deploy to IIS, it works fine.

EDIT: more details about the error:

Server Error in '/' Application.
--------------------------------------------------------------------------------

Could not load file or assembly 'AgileEFLib' or one of its dependencies. An attempt was made to load a program with an incorrect format. 
  Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

 Exception Details: System.BadImageFormatException: Could not load file or assembly 'AgileEFLib' or one of its dependencies. An attempt was made to load a program with an incorrect format.

Source Error: 


 An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.  

Assembly Load Trace: The following information can be helpful to determine why the assembly 'AgileEFLib' could not be loaded.



=== Pre-bind state information ===
LOG: DisplayName = AgileEFLib
 (Partial)
WRN: Partial binding information was supplied for an assembly:
WRN: Assembly Name: AgileEFLib | Domain ID: 3
WRN: A partial bind occurs when only part of the assembly display name is provided.
WRN: This might result in the binder loading an incorrect assembly.
WRN: It is recommended to provide a fully specified textual identity for the assembly,
WRN: that consists of the simple name, version, culture, and public key token.
WRN: See whitepaper http://go.microsoft.com/fwlink/?LinkId=109270 for more information and common solutions to this issue.
LOG: Appbase = file:///C:/src/MyProject/
LOG: Initial PrivatePath = C:\src\MyProject\bin
Calling assembly : (Unknown).
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: C:\src\MyProject\web.config
LOG: Using host configuration file: C:\Users\Shaul\Documents\IISExpress\config\aspnet.config
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Attempting download of new URL file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/Temporary ASP.NET Files/root/15406131/a0c36da/AgileEFLib.DLL.
LOG: Attempting download of new URL file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/Temporary ASP.NET Files/root/15406131/a0c36da/AgileEFLib/AgileEFLib.DLL.
LOG: Attempting download of new URL file:///C:/src/MyProject/bin/AgileEFLib.DLL.
ERR: Failed to complete setup of assembly (hr = 0x8007000b). Probing terminated.