You Had Me At EHLO...

Pacific Northwest Unified Communications User Group - 7/30

2008-07-24T00:15:00Z

Please join us next Wednesday July 30th, 4-7PM PST for the next meeting of the UCDoers - Pacific Northwest Unified Communications User Group meeting!

The UCDoers will meet at the McKinstry Company located at:

5005 Third Avenue South
Seattle, WA 98134
(http://www.mckinstry.com/).

This month Greg Taylor & Per Farney of Microsoft's Certified Architect (MCA): Messaging Program also known as the "Exchange Rangers" will join us. Greg and Per will provide an overview of the program including requirements to attend, what the training program consists of and the benefits of the Exchange Ranger program to the Messaging Community. We will also provide an update on upcoming meetings\plans for the group as we continue to grow the UCDoers User Group!

Come out and join us; food and drinks will be provided!
RSVP: http://upcoming.yahoo.com/event/923349/?ps=5

Thanks,
Rob Herman | Unified Communications Messaging Specialist

Share this post :

Where does the time go? -519 Jet_errLogSequenceEnd

2008-07-23T20:51:00Z

Enterprise Communications Support (formerly known as Exchange Messaging Support) has recently seen an increase in support incidents around the -519 Jet_errLogSequenceEnd issue. In this blog I will explain this issue in detail, why it may occur, how to recognize it, and what to do next.

Exchange has had a robust data engine that provides an ACID database from the early days; the transaction-based JET database engine ensures all database changes are Atomic, Consistent, Isolated and Durable. We accomplish this with the use of transaction log files. In Exchange Server 2000 and 2003, we introduced the concept of Storage Groups. Storage Groups can contain up to 5 databases, but share one set of common Transaction Logs. To differentiate between Storage Groups, all Transaction Log file names contain a prefix, followed by a 5 digit hexadecimal number, like so:

Enn00001.log

Where nn is the number of the Storage Group, typically 01, 02, and so on. As transactions (changes such as new e-mail, mail being read, tasks created, views built, etc) occur Exchange records them in sequentially numbered logs. This allows us to recover from certain database problems by knowing exactly which log comes next in a replay sequence. In Exchange 2000 and 2003, the transaction log file name length is 8 characters long. Since 3 of the characters form the file name prefix, 5 remain. Thus, the largest possible hexadecimal number we can represent is

EnnFFFFF.log

The actual largest number ESE in 2000/2003 will allow is FFFF0. That number in decimal is 1,048,560, representing the maximum number of log files we can write sequentially before running out.

With over one million 5 MB log files available, in some cases, it can take as long as a few years to hit this condition. When the transaction log sequence is exhausted, the Microsoft Jet database engine returns error -519, JET_errLogSequenceEnd to the Information Store. Depending on which version of Exchange 2000 or 2003 you are on, this error will result in slightly different symptoms. These symptoms are described in the following Knowledge Base articles:

830408 Store databases are dismounted without warning or users cannot log on to their mailboxes in Exchange Server 2003 or in Exchange 2000 Server

http://support.microsoft.com/default.aspx?scid=kb;EN-US;830408

896001 An event is not logged in the Application log before the last available transaction log in the sequence is used in Exchange 2000 Server

http://support.microsoft.com/default.aspx?scid=kb;EN-US;896001

In the latest revisions for Exchange 2000 and 2003, we added some fixes to warn you of this impending problem and prevent it from occurring. Store will warn you when you are nearing the end of the log sequence with the following event:

Event Type: Warning
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 514
Description: Information Store (2748) SG2: Log sequence numbers for this instance have almost been completely consumed. To begin renumbering from generation 1, the instance must be shutdown cleanly and all log files must be deleted. Backups will be invalidated.

If you have databases in Storage Groups that have been online contiguously for several years, you should be monitoring your transaction log sequence and the Application Log for this event. When you see this, it's time to reset the transaction log sequence.

Notice that if you miss the ESE 514 warning your databases will dismount and generate the following events:

Event ID: 1159
Event Type: Error
Event Source: MSExchangeIS
Event Category: General
Description: Database error 0xfffffdf9 occurred in function JTAB_BASE::EcEscrowUpdate while accessing the database "First Storage Group\Mailbox Store (SERVER)".

Event ID: 9518
Event Type: Error
Event Source: MSExchangeIS
Event Category: General
Description: Error 0xfffffddc starting Storage Group Path_of_Storage_Group on the Microsoft Exchange Information Store. Storage Group - Initialization of Jet failed.

OK, great, what the heck does 0xfffffddc mean? Glad you asked! You can look up Exchange error codes here: Microsoft Exchange Server Error Code Look-up

Note that error translates to Jet_errLogSequenceEndDatabasesConsistent:

Err 0xfffffddc -
# for hex 0xfffffddc / decimal -548
JET_errLogSequenceEndDatabasesConsistent esent98.h
# /* databases have been recovered, but all possible log
# generations in the current sequence are used; delete all
# log files and the checkpoint file and backup the databases
# before continuing */

If you attempt to mount databases in this condition you will discover another cause for this common error in the Exchange System Manager:

An internal processing error has occurred. Try restarting the Exchange System Manager or the Microsoft Exchange Information Store service, or both.
ID no: c1041724
Exchange System Manager

Let's Get Fixed!

Now that I've explained the issue and how it will be reported in the event logs, here's how to correct this problem:

KB 830408 has a workaround section that describes how to reset transaction log sequence manually. However, we have included this functionality in the Microsoft Exchange Troubleshooting Assistant "Reset log generation number task". We talk about that here:

MSExchangeIS 9518 (0xfffffddc): Exceeded the Maximum Number of Transaction Logs Available for this Storage Group

Microsoft recommends using the Troubleshooting Assistant (ExTRA) because it automates the process of verifying the health of your databases and resetting the transaction log sequence. Download the Microsoft Exchange Troubleshooting Assistant v1.0 and follow these instructions to reset transaction log sequence: http://technet.microsoft.com/en-us/library/aa998611(EXCHG.80).aspx

We also recommend monitoring all your Exchange 2000 and 2003 Storage Groups' transaction log sequences to avoid a potential temporary outage of service. Monitor your application logs for the ESE 514 events.

Now, while it is theoretically possible to hit this condition in Exchange 2007, it will probably take a really long time. For Exchange Server 2007 we decreased the log file size to 1 MB but increased the transaction log file name length to 11 digits, meaning we can go up to

EnnFFFFFFFF.log

Due to the way ESE math works our upward limit is actually 7fffffec log files, but that is still a HUGE number. After figuring in the change in size to 1 MB, that's still about 409 times the number of logs we could generate in 2000/2003. Read more about this improvement in the following TechNet Magazine article located here.

Corbin Meek
Enterprise Communications Support

Share this post :

EWS CAS to CAS Request Proxying

2008-07-18T20:32:56Z

In Exchange Server 2007 SP1, the EWS team added a new feature called EWS Request Proxying. It is sometimes also referred to as EWS Proxy or HTTP Proxy. Not much has been said about this feature, but if you care about performance, it is important to understand why it is there, why you want to avoid it if at all possible, and if you cannot avoid it, why it is important to have.

What is it?

EWS Request Proxying only comes into play in Exchange installations that have multiple Active Directory sites. If you have a single AD site, you can stop reading this post right now. Now that our readership has become smaller, let's continue. Active Directory sites are typically created to define a boundary of highly connected computers and devices. This implies that accessing resources cross-site is more (potentially much more) expensive than accessing resources within a single site. EWS lives on a Client Access Server (CAS) within a specific site. If the mailbox that EWS is trying to communicate with resides on a mailbox server in a different Active Directory site, the ensuing RPC calls between EWS and the mailbox server will be made across a potentially expensive cross-site link. Depending on the EWS operation that is being performed, there may be *many* resulting cross-site RPC calls.

To combat this potentially expensive RPC chattiness, when EWS encounters a request that needs to be serviced in another AD Site, it will actually package the entire request up and make an EWS request to a CAS server in the destination site rather than face cross site RPC calls. The initial CAS box is proxying the inbound HTTP request to a more appropriate CAS server, which is why you will sometimes hear this feature referred to as "HTTP Proxy". In Exchange 2007 SP1, EWS determines the "best CAS" to proxy to by scoring all the items in the request and determining which site can service the most items. The downside of this is that IF your request needs to access mailboxes in multiple AD sites, you WILL encounter cross site RPC calls since EWS does not "split up" requests.

The InternalNLBBypassURL

Although the EWS service is generally stateless, there is some state that is maintained to reduce the amount of Active Directory and Mailbox lookups that EWS has to do when servicing requests. Continued requests from a given user will be quicker and cheaper on a CAS that is maintaining such state. Although any of the CAS servers in the Active Directory site *could* service the request, it is preferable to maintain affinity between the caller and the CAS so that this cached state can be taken advantage of. I will call this "nice affinity" given that it is not a necessity, but rather something that should be preferred if at all possible. In contrast to "nice affinity", there are certain web methods such as GetEvents and Unsubscribe that have very strong CAS affinity given that subscriptions reside in memory on the CAS box where they are created. The Subscribe call itself is governed by the "nice affinity". However, GetEvents and Unsubscribe calls for that created subscriptions must be serviced by that specific CAS box. I will call this "necessary affinity".

When CAS servers sit behind a Network Load Balancer (NLB), it is the NLB that determines which CAS will receive a proxied request that is sent to the NLB's URL. Due to "nice affinity", it is preferable that all requests from a given user be proxied to the same destination CAS box so that the cached information can be taken advantage of. If EWS cannot proxy the call to the CAS maintaining "nice affinity" caches, then all is not lost -- EWS can just choose another CAS in the site which will cause that second CAS to lookup and cache the same information. As a result, EWS takes some state from the inbound call and caller and hashes it across all the available CAS boxes in the destination site. Assuming that the topology in the destination site has not changed, a given caller will always be routed to the same CAS box.

In contrast, if a call has "necessary affinity", EWS cannot fall back to another CAS box, but *must* proxy the request to the applicable CAS box or fail the request. In this case, EWS does not hash the request across CAS boxes, but rather examines the subscription Id directly to determine which CAS box to proxy to. As you can see, whether for "nice" or "necessary" affinity purposes, EWS must be able to bypass the NLB and send proxied requests to the CAS boxes directly.

As part of the EWS proxy feature, the *-WebServicesVirtualDirectory PowerShell cmdlets include a new parameter called InternalNLBBypassUrl which allows you to specify the direct URL to get to a CAS box. When a CAS is configured as part of an NLB, the externalURL or internalURL (depending on the location of the NLB) is set to the Url of the NLB. However, as we have seen , EWS must not use this address for proxying and instead relies on the InternalNLBBypassUrl for proxying. When an Exchange CAS box is installed, the InternalUrl and InternalNLBBypassUrl are both set to the same value - the direct URL for that machine. It is important that IF a CAS is put behind an NLB, you do NOT modify the InternalNLBBypassUrl for that virtual directory to point to the NLB.

EWS will only proxy requests to CAS servers that have the InternalNLBBypassUrl set to a non null value. This also means that if you do *not* want to allow servers to proxy to each other, you can turn off proxying altogether by setting the InternalNLBBypassUrl of all your CAS servers to null.

If you want to see all the InternalNLBBypassUrls for web service virtual directories in your topology, run the following from the Exchange Management Shell:

[PS] D:\Windows\System32>Get-WebServicesVirtualDirectory | foreach {$_.InternalNLBBypassUrl.AbsoluteUri}

To set the InternalNLBBypassUrl for a specific CAS server, run the following Cmdlet:

[PS] D:\Windows\System32>$a = Get-WebServicesVirtualDirectory | where-object {$_.Server -eq "MyServerName"}

[PS] D:\Windows\System32>$a | Set-WebServicesVirtualDirectory -InternalNLBBypassUrl "https://MyServerName.MyDomain"

Why you want to avoid it

When EWS proxies a request from an initial to a destination CAS box, the thread on the initial CAS box is blocked for up to a minute until the request to the destination CAS box returns. As such, you will be using resources on *two* CAS servers in order to fulfill your request. In addition, due to the additional routing, your request will take longer to process than if you talked to the destination CAS box directly. So how can you talk directly to the destination CAS box? You should call AutoDiscover *first* to obtain the correct EWS URL that should be used for the mailbox in question. You do AutoDiscover, right? In fact, if you are dealing with multiple mailboxes, you should call Autodiscover once for *each* mailbox you are trying to access. Of course, it is reasonable to cache this information on the client side. However, note that due to mailbox moves/migration, such cached information may end up getting stale, so don't hold onto it too long.

So when *should* you re-autodiscover? As a general practice, we would recommend you calling autodiscover once per day. In addition, if you encounter any of the following failure response codes, you should re-autodiscover:

ErrorMailboxMoveInProgress
ErrorMailboxStoreUnavailable
ErrorNoDestinationCASDueToKerberosRequirements
ErrorNoDestinationCASDueToSSLRequirements
ErrorNoDestinationCASDueToVersionMismatch
ErrorNoRespondingCASInDestinationSite
ErrorProxiedSubscriptionCallFailure
ErrorProxyCallFailed

When you cannot avoid it

There are times, however, where you cannot avoid HTTP proxy. If the destination site has NO external facing CAS boxes and your client is outside of the corporate network, the only way to access a mailbox on the internal-only site is to have it routed through an externally facing CAS box in another site. And this is precisely why HTTP proxy was added - to allow external access to resources on internal facing sites. Even though the mailbox you are trying to access exists in an Active Directory site with no external facing CAS boxes, you should *still* call Autodiscover to determine the EWS Url to send requests to. Why? Because if there is no external facing CAS box for the site in question, Autodiscover returns the closest and cheapest external CAS.

Rules to Live By

Autodiscover every distinct mailbox you are going to access and cache the EWS Url for a limited amount of time.
Group batched requests by EWS Url. In other words, if two mailboxes return different EWS Urls, do not include both in a single request as it will likely result in cross-site RPCs

Overview of the Autodiscover Service
http://technet.microsoft.com/en-us/library/bb124251(EXCHG.80).aspx

More information on HTTP Proxy and redirection in Exchange
http://technet.microsoft.com/en-us/library/bb310763.aspx

- David Sterling

Share this post :

Exchange Server Documentation Updates - July 2008

2008-07-16T22:29:35Z

The Exchange Server documentation team is pleased to announce updates to the Exchange Server content.

To see what content has changed for Exchange Server 2007 with Service Pack 1, take a look at Exchange Server 2007 Documentation Updates.

To see what content has changed for Exchange Server Analyzer, take a look at Exchange Server Analyzer Topic Updates.

In particular, we would like to highlight the following new or updated topics:

White Paper: Continuous Replication Deep Dive: Continuous replication is a key feature of Exchange Server 2007. Read about the technical details and inner workings of continuous replication technology in this white paper.
White Paper: Planning for Large Mailboxes with Exchange 2007: Learn how to deploy large mailboxes in your environment while also achieving high performance, high availability, and rapid recovery-all at a lower cost than with Exchange 2003.

You can see these articles and other Exchange Server documentation content in the Microsoft Exchange Server TechCenter.

The following downloads are also available for SP1 content:

BTW, all our topics in the Exchange Library have a "Topic Last Modified" date at the top of the topic. And, if you wonder which topics apply to Exchange Server 2007 with Service Pack 1, we now have an "Applies to" tag for Exchange 2007 content.

You can now annotate topics in the Exchange Server 2003 and Exchange Server 2007 documentation. Scroll to the Community Content section at the end of any topic in the Exchange Server Library, and click Add Community Comment. You'll be asked to sign in with your Windows Live ID and to register as a participant. Then, share your insights with the Exchange community.

- Cathy Anderson, Content Release Manager, Exchange User Documentation

Share this post :

iPhone 2.0; Welcome to Exchange!

2008-07-11T15:04:00Z

If you've not heard; Apple released iPhone 2.0 today which includes a software update to the existing iPhones in the market (yes, we mentioned it when it was announced as well). We're thrilled to add them to the family of Exchange ActiveSync licensees that enable all sorts of devices to connect to Exchange Server. For those of you that manage Exchange Servers this means you may see some new devices connecting and we wanted to give you a few notes about what to expect.

What iPhone looks like from an Administrator's perspective

From the server side, you need to look at a user's device from the Exchange Management console:

From the user screen you can scroll though any devices the users has connected to their account (iPhone circled here - note that the version number will vary by iPhone firmware version, we took this screenshot with beta firmware):

Users using OWA will see their iPhones showing up in the Options > Mobile Device screen as shown in the image below:

Note: If you want to look for connections in your IIS server logs you can do a string search for "Apple-iPhone".

How do I find out more info on what policies the iPhone supports, how it connects to your server and other administrative questions?

Apple has published an Enterprise Deployment Guide for organizations that are deploying iPhones. This is where you should look for Administrator info on the technical side of what Apple has created.

How can I see how many iPhones are connecting to my server and which users have them?

To see how many users have iPhones and who they are, go though the following steps:

First you need to open an Exchange Management Shell window and execute the following command:

export-activesynclog -Filename:<IISlog dir>\*.log -outputpath:<output path>

An example is shown below though we just parsed one of the logs for simplicity.

Now open the file Users.csv in Excel. Below you can see the first three columns of this spreadsheet that we're sorted by column C (circled). You can see that by doing this you will be able to see all the iPhones grouped together and their owners will be listed in column A (circled):

What are your experiences?

So now you know what the iPhone will look like connected to your servers using Exchange ActiveSync (instead of IMAP) and how to find out who is using them in your organization. We're glad to have Apple connecting their devices to Exchange Server and hope you have fun using these tools to stay informed about when iPhones connect to your Exchange Server. We're always looking to hear how people are using our technology and we'd love to hear your experiences; are you seeing iPhones show up in your organization? What experiences are your users having? Let us know.

- Adam Glick

Share this post :

TechNet Webcast: Microsoft Exchange Server 2007 Storage Deep Dive

2008-07-10T20:12:31Z

Wanted to let you know about this Webcast as it is happening next week... Ross Smith IV is presenting; you can expect an awesome look into designing Exchange 2007 Storage!

You can find the details and register here:

TechNet Webcast: Microsoft Exchange Server 2007 Storage Deep Dive (Level 300)

- Nino Bilic

Share this post :

Update Rollup 3 for Exchange Server 2007 SP1 and Update Rollup 7 for Exchange 2007 RTM have been released

2008-07-08T21:16:06Z

EDIT 7/9/2008: We have updated the troubleshooting section.

Download information for Update Rollup 3 for Exchange 2007 SP1

The update is live at:
http://www.microsoft.com/downloads/details.aspx?FamilyId=63E7F26C-92A8-4264-882D-F96B348C96AB&displaylang=en

Related KB article:
http://support.microsoft.com/?kbid=949870

Download information for Update Rollup 7 for Exchange 2007 RTM

The update is live at:
http://www.microsoft.com/downloads/details.aspx?FamilyId=086A2A13-A1DE-4B1D-BD12-B148BFD2DAFA&displaylang=en

Related KB article:
http://support.microsoft.com/?kbid=953469

The above update Rollups will also be released to Microsoft update.

Fixes for security issue detailed in MS08-039

A security issue has been identified in Exchange Server 2007 as documented in http://www.microsoft.com/technet/security/bulletin/MS08-039.mspx.

Customers running Exchange Server 2007 RTM need to apply Update Rollup 7 for Exchange 2007 RTM to address the security issue.
Customers running Exchange Server 2007 SP1 need to apply Update Rollup 3 for Exchange 2007 SP1 to address the security issue.

Rollup installation troubleshooting

Seeing that those Rollups contain security fixes, we expect that a lot of people will be applying them. There are two possible issues that we would like you to be aware of:

Exchange 2007 managed services might time out during certificate revocation checks
During the installation of the Rollup, you might encounter a message that you have to wait until the disk space calculation is completed. This message will clear by itself and then you will be able to proceed further. We will permanently resolve this in the future.
When installing a Rollup, we recommend you use the same account that you used to install Exchange Server. If you are using a different account, that account needs to have Local Administrator rights as well as rights to read Active Directory on Exchange object as well as server level (as the update needs to determine which roles are installed on the server). Not having required permissions can lead to OWA not being updated correctly and displaying a blank page after update has completed.

- Nino Bilic

Share this post :

You Had Me At EHLO...

Pacific Northwest Unified Communications User Group - 7/30

Where does the time go? -519 Jet_errLogSequenceEnd

Let's Get Fixed!

EWS CAS to CAS Request Proxying

What is it?

The InternalNLBBypassURL

Why you want to avoid it

When you cannot avoid it

Rules to Live By

Related articles

Exchange Server Documentation Updates - July 2008

iPhone 2.0; Welcome to Exchange!

TechNet Webcast: Microsoft Exchange Server 2007 Storage Deep Dive

Update Rollup 3 for Exchange Server 2007 SP1 and Update Rollup 7 for Exchange 2007 RTM have been released