SolutionBase: Optimizing ISA Server 2004 firewall policies

by Guest Contributor | Mar 14, 2005 7:11:00 PM

Takeaway: ISA Server 2004 can be a powerful firewall, but to make it work properly you must know how to adjust its firewall policies. This article shows how firewall policies work and how to keep them from conflicting.

One of the most challenging aspects of maintaining network and perimeter security is configuring effective firewall policy. Each firewall has its own methodologies and mechanisms for enforcing firewall policy and ISA Server 2004 firewalls are no exception. In order to configure an effective firewall policy, you need to understand how the ISA firewall compares the characteristics of the connection attempt and the rules in the ISA firewall policy. Here's how ISA Server 2004 sees the network world, how it processes firewall policy, and how you can optimize firewall policies on your ISA Server 2004 firewall.

How ISA Server 2004 sees the network world

The first step to understanding how the ISA firewall policy works is to understand the difference between outgoing and incoming requests. Most of us would think of an outbound connection as one that sources from somewhere within the corporate network and has a destination on some Internet location or other untrusted network. We would consider an inbound connection as one sourcing from an Internet location or untrusted location with a destination somewhere on the corporate network.

This was the methodology used by ISA Server 2000 and is the current implementation for many other firewalls. In contrast to how these other firewalls see the network world, the ISA Server 2004 firewall has no concept of trusted or untrusted network nor does it have a concept of internal versus external networks. The ISA firewall sees only Networks. An ISA firewall Network (with a capital N) is one for which the ISA firewall has a definition. As for networks for which the ISA firewall doesn't not have an explicit definition, they are lumped into the ISA firewall's default External Network (with a capital E and capital N).

Secure by design, by default, and in deployment

The ISA firewall does not trust any network, and out of the box the ISA firewall does not allow any connections to it, or through it. It's completely secure, if not very functional. After completing the installation of ISA Server 2004, you will not be able to connect to any resources on the ISA firewall computer and you will not be able to connect to any resources that require traversing the ISA firewall. This is consistent with Microsoft's new philosophy of secure by design, secure by default, and secure in deployment. This is very different from the default configuration of many popular firewalls today, where an allow everything outbound rule is created by default.

Given that the ISA firewall doesn't trust any network and that there are no internal/trusted nor external/non-trusted networks, then what determines an inbound or outbound connection?

Defining inbound and outbound requests

Any connection made through the ISA firewall that uses an Access Rule is considered to be an outbound request. Any connection made through a publishing rule is considered to be an inbound request. Access Rules use protocol definitions where the primary connections are defined as outbound and publishing rules use protocol definitions where the primary connections are inbound.

The key issue here is not the directionality of the protocol definitions so much as the route relationship between the source and destination ISA firewall Network and the level of security and application layer inspection applied to a connection. It's the route relationship and the level of functionality required that will determine whether you create an Access Rule or a publishing rule.

For example, let's suppose you have an ISA firewall and three NICs connected to that firewall. The external interface is directly connected to the Internet and has several public addresses bound to it. The internal interface is connected to the corporate private network and uses private addresses and the third NIC is connected to a DMZ segment has contains a subnet of your public block.

You configure several Network Rules that:

define a NAT relationship between the corporate private network and the Internet
define a Route relationship between the DMZ and the Internet
define a NAT relationship between the corporate private network and the DMZ

If there is a NAT relationship between the source and destination ISA firewall Network, then Access Rules must be created to allow connections from the NATed side to the non-NATed side and publishing rules are created to allow connections from the non-NATed side to the NATed side. There are no exceptions to these rules.

If there is a Route relationship between the source and destination ISA firewall Network, then you can create either Access Rules or publishing rules to control traffic moving from the source and destination networks. You could create an Access Rule to allow incoming SSL traffic from the Internet to the public address DMZ segment, or you could use a Web or Server publishing rule. Your decision to use an Access Rule versus a publishing rule hinges on the level of stateful application layer inspection on the connections moving through the ISA firewall. In general, you'll obtain a higher level of stateful application layer inspection for publishing rules than for Access Rules.

Firewall policy processing for outbound requests

Although it's well known that the ISA firewall processes rules in the firewall policy from the top down, things aren't necessarily that straightforward. There are some peculiarities and contingencies you need to take into consideration when you try to determine how the ISA firewall is making allow and deny decisions.

The Network Rule rules

When an outbound connection is made from a host to a resource through the ISA firewall, the firewall first checks to see if there is a Network Rule that controls the route relationship between the source and destination network. If there is no Network Rule defining the route relationship between the source and destination Network, then the connection attempt is immediately dropped.

You'll likely encounter this situation when you try to configure Internet access for VPN clients. You can configure Access Rules that allow members of the VPN Clients Network to access the Internet through the ISA firewall. This enables you to enforce corporate firewall policy on Internet access even for your VPN clients. However, if you only create an Access Rule allowing the connection, the connection attempt to access the Internet from a host on the VPN Client's Network will fail, because there is no default Network Rule controlling the route relationship between the VPN Clients Network and the Internet.

If a Network Rule exists that defines the route relationship between the source and destination Network, then the ISA firewall begins comparing characteristics of the connection to those defined in the firewall policy's Access Rules, beginning with the first rule and working through the list in order until a rule is found that matches the connection's characteristics.

Connection characteristic checklist

The characteristics of the connection that the ISA firewall checks against the settings in an Access Rule include the following, and are checked in the order listed below:

Protocol
Source address and port
Schedule
Destination address or names or URLs
Users
Content groups

Understanding the order in which characteristics of the connection are checked can give you some insight into which rules are applied and how much processing is required to deny a connection.

For example, suppose you have two Access Rules:

Allow all HTTP traffic from the default Internal Network to the default External network for all users at all times
Allow all HTTP traffic from the default Internet Network to the default External network for authenticated users at all times

What happens when a SecureNAT client on the default Internal Network makes a connection attempt to the Internet through this ISA firewall?

How ISA processes the checklist

The ISA firewall first checks to see if there is a Network Rule defining the route relationship between the default Internal Network and the Internet. The ISA firewall determines there is a NAT relationship and begins processing Access Rules, beginning with rule #1. The ISA firewall first checks the protocol, which is HTTP, so this is a match. The firewall then checks the source address and port, which matches the rule (the source address is part of the default Internal Network and the source ports are all ports unless you define otherwise). If there were no match, then the ISA firewall would move down the list of Access Rules. The ISA firewall then checks the schedule associated with the rule. If the connection is made outside the time for which the rule applies, then the ISA firewall continues to move down the list of rules.

Next, the firewall checks the destination in the connection attempt against the destination listed in the rule, which in this case is the default External Network, which encompasses all public addresses except for those you may define as part of a custom ISA firewall Network. If this destination didn't match, then the firewall would move down the list of rules. The firewall then checks to see if the rule applies to a specific set of users and if the user set in the rule does not match the user credentials in the connection attempt, then the connection will be dropped. Finally, for HTTP connections, content groups defined in the rule are compared with the content requested in the rule. If there is no match, the ISA firewall will move down to the next rule.

If the connection's characteristics match all these settings in the rule, then one more network check is done, but only to determine if there is a Web or Firewall chaining rule that applies to the connection. If there is a Web proxy chaining or Firewall chaining configuration, the ISA firewall will again examine the source and destination of the request and route the connection to the destination based on the parameters of the Web proxy or Firewall chaining rule.

The authentication requirement

A key issue in your ISA firewall's Access Rules is the authentication requirement. Suppose we change the firewall policy by reversing the order of the rules listed above to:

Allow all HTTP traffic from the default Internet Network to the default External network for authenticated users at all times
Allow all HTTP traffic from the default Internal Network to the default External network for all users at all times

Now we have the following situation: A SecureNAT client makes an HTTP connection attempt from the default Internal Network to the Internet. A Network Rule exists that defines the route relationship between the default Internal and External Networks. The protocol matches the protocol defined in rule #1. The source address matches one contained in the definition of the default Internal Network. The Schedule for the rule says that it applies at all times. The destination in the request matches that in the rule.

However, rule #1 applies to authenticated users. You might think that the SecureNAT client connection would be ignored; because it applies to authenticated users and the SecureNAT client obviously cannot authenticate, it does not match the characteristics of the rule and the logical assumption is that ISA firewall would move down the list of rules. This is not the case. If a rule applies to a User Set, and the client cannot authenticate, then the ISA firewall decides that it can't determine what user you are and takes the safe approach and assumes that you are not a member of the set to which the rule applies and denies the connection.

For this reason, we should generally place anonymous rules above rules applying to authenticated connections, because this will prevent this situation. However, keep in mind that this is an issue only if all other characteristics of the rule match the connection attempt before the User component is evaluated.

For example, consider the following rules:

Allow all SMTP traffic from the default Internet Network to the default External network for authenticated users at all times
Allow all HTTP traffic from the default Internal Network to the default External network for all users at all times

As before, a SecureNAT client makes an HTTP connection attempt from the default Internal Network to the Internet. When the ISA firewall intercepts the connection attempt, it first checks for a protocol match. Rule #1 first matches the protocol in the connection attempt with its settings, which apply to the SMTP protocol. Since the protocol specified in Rule #1 doesn't match the protocol in the connection attempt, the ISA firewall stops processing for that rule and moves down to the next rule, which in this case matches all the characteristics of the connection attempt.

Deny rules work in the same way, except that the action is to deny the request instead of allowing it.

Optimizing ISA Server 2004 firewall policy

A reasonable approach to configuring your firewall policy is to order your Access Rules in the following way:

Place anonymous deny rules above all other rules.
Place anonymous allow rules above all rules except anonymous deny rules
Place authenticated deny rules above all rules except anonymous deny and allow rules
Place authenticated allow rules below anonymous allow and deny and authenticated allow rules

Configuring the ISA firewall rule set in this way insures that you will be less likely to create rules that conflict, which could inadvertently allow outbound connections that you didn't want to allow or block outbound connections that you didn't want to block. However, this approach doesn't really take performance into consideration.

Two factors to help you create efficient firewall policy

If you think about how the ISA firewall processes Access Rules, you can create a more efficient and effective firewall policy by just considering two factors:

What are the most frequently used outbound protocols?
From what networks do most connections source?

We can take just these two pieces of information and create more efficient firewall policies, at the same time taking into account the general, four-step approach described above for creating firewall policy.

For example, almost all networks have HTTP as the most common outbound protocol and SMTP is typically the second most frequently used protocol. Let's also assume that most of the outbound network traffic is from the default Internal Network (this parameter will be more specific for your environment, but we'll make this assumption in our current example).

Now consider the following firewall policy:

Allow all HTTP traffic from the default Internet Network to the default External network for members of the domain users group at all times
Allow all HTTP traffic from the DMZ Internal Network to the default External network for all users at all times
Allow all SMTP traffic from the Computer Set containing the company's outbound SMTP relays from the DMZ network to the Internet for all users at all times
Allow all SMTP traffic from the default Internal Network to the default External network for members of the domain admins group at all times
Allow all NNTP traffic from the default Internal Network to the default External network for members of the NNTP users group at all times
Allow all IMAP4S traffic from the default Internal Network to the default External network for all users at all times

In this firewall policy, we put the most frequently used protocols at the top of the list. HTTP is the most frequently used outbound protocol and the most common source network is the default Internal Network. So we put this rule on the top of the list. The second most common source Network is the DMZ network, so we place that right under the first rule.

We've recognized that SMTP is the second most common outbound protocol used on the Network, so we place the SMTP rules below the HTTP rules. We have one outbound SMTP rule applying to anonymous connections and one applying to authenticated connections, so we put the anonymous rule above the authenticated rule.

The NNTP and IMAP4S protocols are much less commonly used, so we put those below the HTTP and SMTP rules. By ordering the rules in this way, the ISA firewall doesn't have to process a long list of Access Rules that rarely apply to connections. Depending on how busy your ISA firewall is, you can save the firewall from having to match connections to rules that rarely apply millions to billions of times per month.

0 comment(s)